Evaluation of the Action Plan to Protect Human Health from Environmental Contaminants 2008-2009 to 2012-2013

Prepared by
Evaluation Directorate
Health Canada and the Public Health Agency of Canada

February 2014

Table of Contents

• Executive Summary
• Management Response and Action Plan
• 1 Evaluation Purpose
• 2 Background and Context
  • 2.1 The Action Plan to Protect Human Health from Environmental Contaminants
  • 2.2 Partners and Stakeholders
  • 2.3 Resources
  • 2.4 The Action Plan's Outcomes
  • 2.5 Description of the Results Chain for the Action Plan
  • 2.6 Scope of the Evaluation
  • 2.7 Evaluation Issues and Questions
  • 2.8 Evaluation Methodology
  • 2.9 Limitations and Mitigation Strategies
• 3 Findings - Relevance
  • 3.1 Continued Need
  • 3.2 Link to Government Priorities and Departmental Strategic Outcomes
  • 3.3 Appropriateness of the Federal Government's Role and Responsibility
• 4 Findings - Performance
  • 4.1 Intended Immediate Outcomes
  • 4.2 Intended Intermediate Outcomes
  • 4.3 Intended Longer-term Outcome
  • 4.4 Enabling/Detracting Factors
  • 4.5 Unintended Impacts
  • 4.6 Efficiency and Economy
• 5 Evaluation Conclusions and Recommendations
  • 5.1 Conclusions
  • 5.2 Recommendations
• Appendix A - Evaluation Matrix
• Appendix B - CHMS Evaluation - Management Response and Action Plan
• Appendix C - Detailed Description of Action Plan Component Programs
• Appendix D - Listing of Chemicals Monitored by the CHMS Cycle 1 and FNBI
• End Notes

Executive Summary

The purpose of this evaluation was to assess the relevance and performance (effectiveness, efficiency and economy) of the Action Plan to Protect Human Health from Environmental Contaminants (Action Plan). The evaluation was undertaken to fulfill the requirements of the Financial Administration Act and the Treasury Board of Canada's Policy on Evaluation (2009). The methodology used in the evaluation included document, literature, and administrative and financial data reviews, key information interviews, and a survey of primary data users.

The Action Plan is a federal initiative led by Health Canada (HC) in partnership with Statistics Canada and the Public Health Agency of Canada (PHAC), designed to protect the health of Canadians, particularly vulnerable populations, from harmful contaminants. It received $84.6M in funding over the period 2008-2009 to 2012-2013, and $18.9M in ongoing funding.

Four programs were funded under the Action Plan:

• Development and dissemination of environmental health guides (EHGs) -- $13.1M to HC ($0.8M ongoing)
• An extension of the survey infrastructure at Statistics Canada for the Canadian Health Measures Survey (CHMS) -- $54.5M to Statistics Canada ($14M ongoing)
• Development and implementation of the First Nations Biomonitoring Initiative (FNBI) -- $5.6M to HC ($0.7M ongoing^{Footnote 1})
• Enhanced surveillance of congenital anomalies and surveillance of developmental disorders -- $11.4M to PHAC ($3.4M ongoing)

The EHGs form the 'public awareness' component of the Action Plan, which aims to foster awareness of the risks associated with environmental contaminants and the measures Canadians can take to minimize their exposure. The CHMS, FNBI and PHAC surveillance activities form the 'monitoring and surveillance' component, which aims to increase the evidence base through monitoring and surveillance of the extent of, and trends in, exposure to environmental contaminants and the potential association with adverse health effects.

Findings - Relevance

There is a continuing need for federal efforts to protect Canadians' health from environmental contaminants. There are links between certain environmental contaminants and adverse health effects, and there is an economic and social burden attributed to negative health effects that are linked, at least in part, to environmental causes. Further, risks from exposure and potential negative health effects of many substances are not fully understood. Efforts to increase public awareness about environmental health risks and mitigation strategies benefit Canadians and help address concerns about risks to their health from environmental contaminants. Monitoring and surveillance address gaps in the evidence base, and trend data contributes to assessing the effectiveness of risk management actions.

The Action Plan components are consistent with priorities in the area of environmental health articulated in the 2007 federal Speech from the Throne and subsequent budget announcements. The monitoring and surveillance activities funded by the Action Plan contribute to fulfilling Canada's obligations as signatory to a variety of international agreements pertaining to health and the environment. All components of the Action Plan support the federal government's broader agenda on the environment and health. The Action Plan is linked to two key federal initiatives - the Chemicals Management Plan (CMP) and the Clean Air Agenda (CAA).

The Action Plan is aligned with HC, Statistics Canada and PHAC departmental mandates, and there is concurrence between the Action Plan components and the strategic outcomes and priorities of the three partner departments.

The Action Plan is consistent with federal roles and responsibilities in the area of health and contributes to fulfilling authorities laid out in legislation such as the Canadian Environmental Protection Act, the Pest Control Products Act, the Food and Drugs Act, and the Canada Consumer Product Safety Act, as well asthe federal government's responsibilities with respect to the health of First Nations. The federal role is also consistent with the expectations of stakeholders who have called for a greater commitment to monitoring and surveillance of environmental contaminants, and reflects international practice, including other jurisdictions' national health measures surveys and implementation of national action plans on environmental contaminants.

Findings - Performance

Effectiveness

Public Awareness

The public awareness component of the Action Plan comprised a variety of tactics which included the development and distribution of the mainstream "Hazardcheck" EHG, First Nations and Inuit EHGs, web content, advertising campaigns, promotional activities and outreach.

In terms of the Hazardcheck EHG, the evaluation found evidence that Canadians are aware of connections between environmental exposures and health, although targeted increases in awareness could not be fully assessed due to available data. The distribution of the EHGs exceeded performance targets by a wide margin, and the 'call to action' within the Hazardcheck guide appeared to motivate individuals to take some of the steps recommended in the guide. Additionally, the response to the guide in this area compares favourably with other HC campaigns. Although advertising activities drove a high number of website visits and uptake of the guides, it appeared that recall rates for the first series of ads were below federal baseline values compared to similar campaigns. Subsequent mainstream advertising tactics were adjusted to include only those tactics that had proven to be the most effective.

For the First Nations and Inuit EHGs, formal baseline POR was conducted with First Nations and Inuit prior to developing campaigns for these audiences. Follow up POR (2013) found that awareness of environmental issues continued at levels found in 2010, however, in 2013, individuals were more able to identify, without prompting, some form of environmental issues that affect health. As well, both First Nations and Inuit indicated an increased sense of confidence in taking steps to protect themselves from these environmental risks. The proportion of respondents who felt the environment affects their own or their family's health "a great deal" increased among First Nations on reserves between 2010 and 2013, but decreased among Inuit for the same time period.

The CHMS and FNBI also made contributions to public awareness through interactions and sharing of biomonitoring results with individual participants and, where relevant, participating communities, as well as through publically available reports.

Monitoring and Surveillance

The evaluation evidence indicates that the CHMS and FNBI have contributed to an increase in reliable and usable data with respect to understanding the connections between environmental contaminants and human health. Data collection for two cycles of the CHMS and baseline data for the FNBI have been completed and include measures to assess exposures to a selection of environmental contaminants. Both surveys are based on a comprehensive methodological approach that mirrors international practice in this area, and the FNBI replicates the CHMS methodology to allow for comparisons at the population level. CHMS data users were positive about data quality.

PHAC has completed or initiated various activities related to the enhancement of the congenital anomalies (CA) surveillance system and the creation of the developmental disorders (DD) surveillance system. Limited enhanced data on CA is available and the DD system has not yet progressed to the state of actively collecting data from provinces and territories. Delays were primarily a result of unanticipated challenges associated with the complex nature of disorders such as ASD, and the need to coordinate systems with health and non-health sectors. The DD system is now in its implementation phase and plans to collect population level data by 2015.

The awareness and accessibility of monitoring and surveillance data are mixed. For the CHMS, the data are well known and used by a wide variety of federal researchers and policy makers. However, data accessibility issues for the CHMS were identified for external researchers.^{Footnote 2} For the FNBI, accessibility of the data is currently limited as national level analysis and reporting were only recently carried out (June 2013).

There is preliminary evidence that the CHMS data are being used to inform decision-making in a variety of areas, and there is potential for these data to contribute to reducing the risks to Canadians from environmental contaminants through an enhanced evidence base.

Efficiency and Economy

The perception among key informants was that the Action Plan represented good value for money and was delivered in an efficient manner, utilizing appropriate management, planning and partnerships to realize efficiencies. The Action Plan, as an overarching funding mechanism, did not appear to confer any significant efficiencies or inefficiencies in the delivery of the components. While the four components are linked conceptually, they unfolded quite independently. An exception was the linkage between the FNBI and the CHMS, which served to increase the efficiency of the implementation of the FNBI (by capitalizing on the experience of the CHMS) and the comparability of the FNBI and CHMS data. Action Plan funding, including the provision of ongoing funding, have stabilized the CHMS infrastructure, which was noted to have conferred some efficiencies and enabled Statistics Canada to continue with additional data collection cycles to address ongoing priorities in a variety of health areas. However, parallel FNBI data collection does not appear to be similarly funded. Other operational efficiencies were also identified, for example, the use of social marketing and partnership arrangements.

Analyses of available financial data show significant variances between planned and actual expenditures related to surveillance activities and the EHGs in the early years of the programs. Where funds were unspent on Action Plan activities, information was not always available to determine where within departments they had been transferred or to confirm transfer approvals. As the programs matured, these variances have generally decreased. CHMS and FNBI expenditures show relatively minor variances.

As the components move forward with ongoing funding under the Action Plan, there may be a more compelling rationale for a closer relationship among the components. For example, as monitoring and surveillance activities mature, they may be able to offer greater insights for public awareness messaging. Lessons learned from public awareness activities for the general public and for First Nations and Inuit could also be shared to improve practices while recognizing the unique needs of each target group. Greater use of regional partnerships and networks to capitalize on existing relationships was also noted as a potential area for greater coordination.

RECOMMENDATIONS

This evaluation resulted in two recommendations. The Statistics Canada component evaluation included recommendations for the CHMS (see Appendix C).

Financial Tracking

The evaluation's ability to assess whether program outputs were produced efficiently, or whether expected outcomes were produced economically was hampered by limitations in the data on expenditures. Obtaining complete and consistent financial data proved to be a challenge for some Action Plan components.

Recommendation 1:

All Action Plan partners should implement effective financial data tracking and monitoring processes to ensure financial accountability and facilitate future assessments of efficiency and economy, and to enable Health Canada, as the lead for the Action Plan, to develop annual financial summaries.^{Footnote 3}

Public Health Surveillance

The Public Health Agency has the responsibility for undertaking surveillance of congenital anomalies (in coordination with the Canadian Congenital Anomalies Surveillance Network) and developmental disorders. This includes data collection, analysis and reporting, as well as dissemination of findings. These activities are dependent on partnerships with provinces and territories. Considering the complex nature of disorders such as Autism Spectrum Disorders (ASD), there is a need to coordinate systems with health and non-health sectors.

Evaluation evidence indicates that the Public Health Agency's developmental disorders surveillance system (initially focused on ASD in children and youth) has not yet progressed to the state of actively collecting data from provinces and territories. While progress has been made in developing this system, data is not yet available. The enhancement of the congenital anomalies surveillance system is in a similar state.

Surveillance information is still needed to address gaps in the evidence base and to increase understanding of the occurrence and patterns about congenital anomalies and developmental disorders such as autism.

Recommendation 2:

The Public Health Agency should work with partners to ensure that timely and usable data is available for decision makers on congenital anomalies and autism spectrum disorders, linking the influence of environmental contaminants on these areas.

Canadian Health Measures Survey

The Statistics Canada component evaluation of the CHMS included three recommendations. These recommendations were addressed through a Management Response and Action Plan (see Appendix C).

Management Response and Action Plan

1 Evaluation Purpose

The Action Plan to Protect Human Health from Environmental Contaminants (Action Plan) is a federal initiative to protect the health of Canadians from environmental contaminants by fostering awareness of the risks associated with exposure to contaminants (and actions to minimize these risks), and increasing the knowledge base about contaminant levels and potential impacts on health. The initiative is led by Health Canada (HC) in partnership with Statistics Canada and the Public Health Agency of Canada (PHAC).

The purpose of the Evaluation of the Action Plan to Protect Human Health from Environmental Contaminants was to assess the relevance and performance of the program in fulfillment of the requirements of the Financial Administration Act and the Treasury Board Policy on Evaluation (2009). It also provides insight into the contribution of the Action Plan to other federal initiatives related to health and the environment.

2 Background and Context

2.1 The Action Plan to Protect Human Health from Environmental Contaminants

The Action Plan is a tri-departmental initiative led by HC in partnership with Statistics Canada and PHAC, designed to protect the health of Canadians, particularly vulnerable populations, from harmful contaminants. It received $84.6M in funding over the period 2008-2009 to 2012-2013, and $18.9M in ongoing funding. There are two broad thrusts of the Action Plan: Environmental Health Guides -- to foster awareness of the risks associated with environmental contaminants and the measures Canadians can take to minimize their exposure; and Monitoring and Surveillance -- to increase the evidence base through monitoring and surveillance of the extent of and trends in exposure to environmental contaminants and potential association with adverse health effects. The Action Plan is also intended to complement and support other related government strategies such as the Chemicals Management Plan (CMP) and the Clean Air Agenda (CAA).

2.1.1 The Action Plan Components

Four programs were funded under the Action Plan:

• Development and dissemination of an environmental health guide, modelled after "Eating Well with Canada's Food Guide", to describe the actions that Canadians should take to protect themselves and their families from environmental health risks ($13.1 million/5 years). The objective of the environmental health guide (EHG, known as Hazardcheck), and companion documents (including fact sheets for subpopulations such as seniors, students, teachers), is to make Canadians aware of the risks that harmful environmental exposures may pose to their health, along with the direct actions they can take to reduce these risks and improve their health. EHGs were also developed by the HC First Nations and Inuit Health Branch (FNIHB) for First Nations and for Inuit, including guides for indoor environments, seasonal guides for outdoor environments, youth guides and activity booklets for children and teens. Hazardcheck and the First Nations and Inuit EHGs^{Footnote #edN1} are available in both print format and on the internet.
• An extension of the survey infrastructure at Statistics Canada for the Canadian Health Measures Survey (CHMS) to allow the tracking of changes in health and risk factors, including but not limited to, exposure to environmental contaminants and expansion of the survey to include children under the age of six ($54.5 million/5 years).^{Footnote 5} The CHMS is a national survey that was launched in 2007 to address long-standing limitations and data gaps in Canada's health information system. The CHMS is being used to collect information from Canadians about their general health and lifestyles and provides data for several branches within HC and PHAC. Through interviews and the collection of a range of physical measurements and biological samples (blood and urine), the survey will help estimate the number of Canadians potentially at risk of developing certain diseases. It also enables relationships between disease risk factors and health status to be determined, emerging public health issues to be explored and new measurement technologies to be evaluated so that future surveys can be more effectively designed. The Action Plan provided funding for the CHMS infrastructure for the conduct of subsequent cycles of the survey, expanded data collection in these cycles to include children under the age of six, and ongoing stewardship of biological samples. The Action Plan contributes about 75 per cent of the funding for the CHMS, the remainder of which is covered through cost-recovery from other HC branches, PHAC and other entities.^{Footnote #edN2}
• Development and implementation of the First Nations Biomonitoring Initiative (FNBI)to address the lack of nationally representative data on exposure to environmental chemicals for First Nations living on reserves ($5.6 million/5 years). Action Plan funding allowed for the FNIHB, in partnership with the Assembly of First Nations (AFN), to consult with First Nations, design, develop, and conduct the health survey, and analyze and report on results to individuals, First Nations communities, and the general public.
• Enhanced surveillance of congenital anomalies and surveillance of developmental disordersto increase understanding of the occurrence and patterns through congenital anomalies surveillance and surveillance of developmental disorders ($11.4 million/5 years), and to understand the potential links between environmental contaminants and these health effects.
  • Under the Action Plan, PHAC is working to use the existing congenital anomalies surveillance system to strengthen provincial and territorial surveillance systems with respect to congenital anomalies.
  • For developmental disorders surveillance, the current focus is the development of the autism spectrum disorders (ASD) surveillance system. Canada has no comprehensive system in place to monitor ASD and other developmental disorders. In context, ASD is a complex condition that does not have defined biological markers to enable definitive diagnosis. The clinical definitions of ASD have changed over time, including recent changes announced in April 2013. Further, the indicators of interest for ASD surveillance are not all captured in health databases, which contributes to the need to coordinate systems with health and non-health sectors. Challenges associated with these factors include: the need to pool multiple data sources, given the absence of a single repository of proposed data elements; the lack of uniformity across jurisdictions for the identification of ASD; and, a varied state of P/T readiness for data capture and capacity.

More detailed program descriptions of each of the components of the Action Plan are included in Appendix D.

2.2 Partners and Stakeholders

2.2.1 Partners

The Action Plan is delivered by HC, Statistics Canada and PHAC. Federal partners' roles in the Action Plan are described below.^{Footnote #edN3}

• The Healthy Environments and Consumer Safety Branch (HECSB) at HC is the lead for the Action Plan. In addition, HECSB is responsible for developing the content for Hazardcheck, as well as creating companion guides and fact sheets for specific populations. Subject matter experts within the Branch (e.g., on lead, radon) are partners in content development and important resources for quality assurance.
• FNIHBat HC plays two roles in the Action Plan. The first is to create the EHGs for First Nations and for Inuit, in consultation with First Nations and Inuit partners. The second role of the branch is to work with First Nations organizations (such as the AFN) and communities to design and implement the biomonitoring survey, analyze the data, and disseminate community and national reports on survey findings.
• Public Affairs Communications and Consultations Branch (PACCB)^{Footnote 6} at HC is responsible for producing, branding, marketing and disseminating both the Hazardcheck guide developed in collaboration with HECSB and the First Nations and Inuit EHGs and fact sheets developed in collaboration with FNIHB. They are also responsible for monitoring the uptake of these guides immediately after the marketing campaign.
• Regions and Programs Bureau (RAPB) at HC is involved in the development and dissemination, including outreach, engagement and distribution through regional networks and events, of the environmental health guide "Hazardcheck".
• Statistics Canada is responsible for implementing the extension to the CHMS, which entails the collection of physical measurements and information from Canadians about their general health and lifestyles as well as biological samples (blood and urine) to measure the level of environmental contaminants. Statistics Canada is responsible for disseminating these data through files, fact sheets, and at conferences.
• PHAC has responsibility for undertaking surveillance of congenital anomalies (coordinating this work with the Canadian Congenital Anomalies Surveillance Network) and developmental disorders. This includes data collection, analysis and reporting, as well as dissemination of findings.

Each identified area within HC, Statistics Canada and PHAC is responsible for the management of the initiative(s) they are leading. Governance and management of the Action Plan takes place through existing consultative structures (intra- or interdepartmental committees) that have been established within or across these departments for related purposes (e.g., the Healthy Living and Chronic Disease Initiative of PHAC). The existing tri-partite (HC, PHAC and Statistics Canada) Canadian Population Health Statistics Program is used to oversee the CHMS extension of scope.

2.2.2 Stakeholders

The public awareness component of the Action Plan is intended to reach a broad audience. The Hazardcheck guide is targeted to the general public and EHGs were developed for First Nations and Inuit. Some content within the EHGs is targeted specifically to families and children. For monitoring and surveillance, specific groups have a particular interest in the outputs of the Action Plan. Federal (e.g., other branches within HC, Environment Canada, Canada Mortgage and Housing Corporation) and provincial decision-makers are expected to find the monitoring and surveillance outputs useful in determining chemical, environmental, industrial, and consumer product regulations, policies, and guidelines. The reach of the activities under the Action Plan extends to a range of other stakeholders and beneficiaries that include:

• Organizations and institutions supported by provincial/territorial governments, such as regional health authorities, public health units, and educational institutions and hospitals;
• Municipal governments;
• Not-for-profit, voluntary, and industry organizations (national, regional, provincial/territorial, and local, including First Nations and Inuit organizations);
• Academics and researchers;
• Health professionals and para-professionals, and professional associations; and
• Special interest groups/stakeholder associations and organizations.

The ultimate reach of the Action Plan is Canadians who benefit from actions taken to protect their health from harmful environmental contaminants.

2.3 Resources

The Action Plan was allocated $84.60M over five years. This included: $18.7M over 5 years (including $3.98M in grants and contributions) and $1.5M ongoing to Health Canada; $11.40M over 5 years (including $1.15M in grants and contributions^{Footnote 7}) and $3.4M ongoing to PHAC; and, $54.50M over 5 years and $14M ongoing to Statistics Canada.

2.4 The Action Plan's Outcomes^{Footnote 8}

In terms of the Environmental Health Guides, in the immediate term, activities are expected to lead to increased public awareness of connections between environmental exposures and health. The achievement of this immediate outcome is expected to lead to the intermediate outcome of
increased action by Canadians to minimize environmental health risks.

In the immediate term, activities related to the CHMS, FNBI and PHAC surveillance systems are expected to lead to reliable and useable data for decision makers, researchers and Canadians. This, in turn, is expect to lead to increased data/information user awareness of the collected data/information, and increased use of the information on the associations between contaminants and illness to guide decision-making in public health practices, research, policy, regulation, programs and services.

Ultimately, the Action Plan aims to lead to reduced health risks to Canadians (particularly vulnerable populations) from harmful environmental contaminants.

2.5 Description of the Results Chain for the Action Plan

There are three main streams of activities contributing to the Action Plan's ultimate outcome of reduced health risks to Canadians (particularly vulnerable populations) from harmful contaminants. These include: 1) the Environmental Health Guide, 2) First Nations Biomonitoring and the Canada Health Measures Survey, and 3) Surveillance.

1. Environmental Health Guide

HECSB undertakes the activities below:

• Develop main guide, tailored versions (First Nations and Inuit), companion guides, and fact sheets (teachers, students and seniors)

These activities are expected to lead to the following outputs:

• Environmental Health Guide for Canadians
• Tailored guides for First Nations & Inuit
• Companion Guides & Fact Sheets for teachers, students, seniors, etc.

PACCB undertakes the activities below:

• Market and disseminate main guide, tailored versions and companion guides
• Brand product
• Monitor take-up

These activities are expected to lead to the following outputs:

• Marketing campaign

Together, the HECSB and PACCB activities and outputs are expected to result in the immediate outcome of 'increased public awareness of connections between environmental exposures and health, and the intermediate outcome of increased action by Canadians to minimize environmental health risks'.

2. First Nations Biomonitoring and Canada Health Measures Survey

FNIHB undertakes the activities below:

• Establish agreements with First Nations operations
• Develop & implement survey
• Collect & analyze data
• Disseminate data

These activities are expected to lead to the following outputs:

• Agreement with First Nations organizations
• Survey
• Biomonitoring data
• Reports on data

These FNIHB outputs are expected to lead to the immediate outcome of 'reliable and usable data for decision makers, researchers & Canadians on the baseline health status of First Nations & exposure to environmental contaminants that will enable comparison between First Nations and the Canadian population'.

Statistics Canada undertakes the activities below:

• Collect health data on Canadians through health measures survey
• Disseminate data in tables, files & fact sheets & at conferences

These activities are expected to lead to the following outputs:

• Data on health measures
• Reports on data
• Dissemination plan

These Statistics Canada outputs are expected to lead to the immediate outcome of 'reliable and usable data for decision makers, researchers and Canadians on the baseline health status of Canadians, level and exposure to environmental contaminants'.

3. Developmental Disorders Surveillance and Congenital Anomalies Surveillance

PHAC undertakes the activities below to establish a surveillance system for development disorders:

• Establish capacity within PHAC
• Establish conditions to be monitored
• Establish Governance
• Assess external capacity (in provinces/territories) & establish sentinel sites in select provinces/territories
• Analyse surveillance data
• Disseminate surveillance results

These activities are meant to lead to the following outputs:

• Sentinel sites in select provinces/territories
• Data and reports on developmental disorders occurrence & trends & the influence of environmental contaminants
• Dissemination plan

PHAC undertakes the activities below to enhance the surveillance system for congenital anomalies:

• Develop partnership agreements with provinces/territories
• Analyze surveillance data
• Disseminate surveillance results

These activities are meant to lead to the following outputs:

• Agreements with provinces/territories
• Data and reports on congenital anomalies occurrence & trends & the influence of environmental contaminants
• Dissemination plan

The outputs from both PHAC areas are expected to lead to the immediate outcome of 'reliable and usable surveillance information on congenital anomaly occurrence trends, developmental disorder trends and the influence of environmental contaminants for decision makers'.

Together, the FNIHB, Statistics Canada and PHAC immediate outcomes are expected to result in the intermediate outcome of 'increased data/information user awareness of the collected data/information'. This, in turn, should result in the long-term outcome of 'decision makers increasingly use the information on the associations between contaminants and illness to guide decision-making in public health practice, research, policy, regulation, programs & services'.

2.6 Scope of the Evaluation

The evaluation includes the four component programs funded under the Action Plan:

• The Health Canada (HC) environmental health guides (EHGs);
• The HC First Nations Biomonitoring Initiative (FNBI);
• The Statistics Canada Canadian Health Measures Survey (CHMS); and
• The Public Health Agency of Canada (PHAC) congenital anomalies and developmental disorders surveillance systems.

Statistics Canada conducted a component evaluation of the CHMS, and PHAC conducted an evaluation data collection exercise for the congenital anomalies (CA) and developmental disorders (DD) surveillance systems. Findings from these documents have been rolled-up with data pertaining to the EHGs and FNBI collected as part of this evaluation. This evaluation was conducted by EKOS Research Associates, an independent evaluation consulting firm, on behalf of Health Canada, Statistics Canada and PHAC.

The evaluation was guided by an Umbrella Results-based Management and Accountability Framework (RMAF) for the Action Plan (March 2008) and a Roll-Up Evaluation Framework for the Action Plan (December 2009). The scope of the evaluation is from 2008-2009 to 2012-2013. For the CHMS, the scope was expanded by one year to include 2007-2008 to capture CHMS's Cycle 1, which was launched in 2007. This was done because this was the only cycle of the survey that was fully completed at the time of the evaluation. The Action Plan represents an initial investment in long-term efforts to generate monitoring and surveillance data and create public awareness and action to minimize risk from environmental contaminants. As a result, the focus of the evaluation was at the immediate outcome level.

2.7 Evaluation Issues and Questions

The evaluation of the Action Plan focuses on the core issues listed in the Directive on the Evaluation Function accompanying the 2009 Treasury Board Policy on Evaluation. As per the policy, the evaluation examines the relevance and performance (effectiveness, efficiency and economy) of the Action Plan. The evaluation issues and questions in Table 2 guided the development of the data collection instruments and the collection of data for each of the four component programs. The complete matrix of evaluation questions, indicators and data sources, including targets that were set for selected indicators, is presented in Appendix B.

2.8 Evaluation Methodology

The evaluation of the Action Plan is based on multiple lines of evidence. The approach to evaluation planning and the evaluation methods are described below.

2.8.1 Evaluation Approach

The evaluation used a goals-based approach to determine, at the end of a five-year period, to what extent the intended outcomes of the Action Plan were achieved. As mentioned above, the Statistics Canada component evaluation of CHMS and the PHAC evaluation data collection exercise have been used as data sources for this roll-up evaluation. Evaluation methodologies for the supporting evaluation activities were designed to avoid duplication with Action Plan roll-up evaluation activities, and to avoid response burden in contacts with key informants who may be involved in more than one of the monitoring and surveillance components.

2.8.2 Evaluation Methods

This evaluation incorporates a combination of qualitative and quantitative measures to ensure a balanced analysis of relevance and performance of the Action Plan. The table below summarizes the methodologies that were used for each of the Action Plan components.

Each of the data collection methodologies is summarized below. The analysis and triangulation of findings was conducted following the data collection activities. For each evaluation question, the findings from each of the relevant data sources and from the Statistics Canada and PHAC evaluation documents were identified, summarized and triangulated to draw substantiated findings and conclusions.

2.9 Limitations and Mitigation Strategies

The evaluation encountered a number of data gaps in addressing the evaluation questions related to performance. Gaps in public opinion research (POR) for the mainstream Hazardcheck guide made it difficult to ascertain changes in awareness and behaviour as a result of the mainstream program. When POR data were not available, other information sources were utilized such as EHG distribution statistics and evaluations of specific marketing campaigns (e.g., the Hazardcheck retail-based public engagement campaign). Because both baseline and follow up POR was conducted with First Nations and Inuit audiences, there were no data gaps in assessing changes in awareness and behaviour amongst this group.

There was a second challenge in assessing cost-efficiency and effectiveness of the program components, as expenditure data were difficult to obtain for a variety of reasons. Within the FNIHB, a common financial code for the First Nations and Inuit EHGs and for the FNBI made it difficult to disentangle the unique expenditures for each component. For the CHMS program, the practice of compiling financial data by fiscal year and not by cycle, the lack of baseline data on cost and resources used by level output produced, and the lack of efficiency data on similar surveys were impediments to a cost-efficiency analysis. To partially compensate for these gaps, allocated funding was used in lieu of expenditures to describe program resources. However, the analysis of cost-efficiency of the Action Plan components is limited.

The survey of primary data users for the CHMS had a low response rate (22 per cent) (n=30), which may have been due to a comparatively short two-week window for fielding the survey. The views of data users were also canvassed through key informant interviews conducted for the CHMS evaluation and for the roll-up evaluation. Survey evidence was also triangulated with other findings (e.g., from the document review) to address the evaluation questions.

Finally, although a very specific performance measurement framework was outlined in the Action Plan for the partner departments, there is limited evidence that performance data were systematically collected on an ongoing basis according to this framework. This was due, at least in part, to delays in the implementation of some components and the data gaps referred to above. The evaluation utilized performance information where it was available and has noted where evidence gaps or implementation delays limit the assessment of some intended outcomes.

3 Findings - Relevance

This section on program relevance presents the evaluation findings on the continued need for the Action Plan, the alignment of the Action Plan and component programs with federal and departmental priorities, and the extent to which the Action Plan is consistent with federal roles and responsibilities.

3.1 Continued Need

3.1.1 Links between environmental contaminants and health

Finding 1. There are demonstrated links between certain environmental contaminants and adverse health effects, which carry a measurable economic and social burden.

A review of documentation and literature supports the continued need for federally led efforts to protect human health from environmental contaminants in Canada. There is a significant body of literature on the negative impacts of certain environmental contaminants, both synthetic and natural, on human health. Substances such as lead, mercury, and polychlorinated biphenyls (PCBs) have been found to pose risks at certain levels of exposure, particularly for susceptible groups such as children and pregnant women. Some substances have been linked to adverse health effects such as respiratory, cardiovascular, neurological and reproductive problems, as well as some cancers.^{Footnote #edN4}

Literature reviewed for the evaluation of the PHAC surveillance systems suggests that the etiology of congenital anomalies and developmental disorders such as Autism Spectrum Disorders (ASDs) is complex and is largely multifactorial, i.e., caused by the interaction of genetic and environmental risk factors.^{Footnote #edN5}

A substantial part of the total burden of disease in industrialized countries has been attributed to environmental factors.^{Footnote #edN6} In Canada, 2004 World Health Organization (WHO) data indicate that approximately 13 per cent of the total burden of disease has environmental causes.^{Footnote #edN7} Estimates of the proportion of cases of congenital anomalies due to environmental factors (maternal-related conditions, drug or chemical exposures) range between two and twelve percent.^{Footnote #edN8},^{Footnote #edN9}

Within the Canadian context, vulnerable populations include Aboriginal people due to increased exposure to environmental contaminants such as persistent organic pollutants (POPs), PCBs, and heavy metals like lead and mercury, compared to the general population.^{Footnote #edN10}

The economic and social costs of disease attributable to environmental factors are significant. For example:

• In HC's Risk Management Strategy for Lead (2013), the socio-economic burden caused by early childhood lead exposure in Canada is estimated as the present value of foregone lifetime earnings resulting from reduced intellectual development. The estimated range of the impact is from $1.5 billion to $9.4 billion per year.
• According to an extensive scoping review of existing research, cited in the PHAC evaluation report, the annual disease burden of congenital anomalies attributable to environmental exposures in Canada has been estimated as follows: 72 to 360 deaths; 128 to 640 serious congenital anomalies; 300 to1500 hospitalizations; 2000 to 10,000 patient-days spent in hospital; and 500 to 2500 low birth weight babies.^{Footnote #edN11} In the US, for persons with an intellectual disability, the Centers for Disease Control and Prevention (CDC) estimates the direct and indirect costs per person over a lifetime to be over $1 million.^{Footnote #edN12}

3.1.2 Canadians' awareness of the links between environmental contaminants and health

Finding 2. Canadians are aware of, and express concern about potential negative health effects from environmental contaminants.

Public opinion research (POR) indicates that Canadians are aware of the potential for negative impacts on health from contaminants in the indoor and outdoor environments, including toxic chemicals. According to POR^{Footnote 8} conducted in 2007 (prior to the Action Plan), for example, 63 per cent of Canadians believe that environmental problems now affect their health 'a great deal' or 'a fair amount'.^{Footnote #edN13} In 2010, POR^{Footnote 9} conducted by HC on chemicals management and environmental health issues found that three in ten Canadians indicate that they or someone else in their family has experienced a health problem that was either clearly or possibly the result of exposure to a chemical.^{Footnote #edN14} About one in ten Canadians name chemicals as posing the greatest risk to human health and indicated greater concern about chemicals in the environment than in their home. Respiratory problems and cancer are cited most often as potential health impacts of exposure to toxic chemicals. Few respondents could recall without prompting any other potential health impacts of toxic chemicals.

POR conducted in 2010^{Footnote 10}, while focused on Canadians' awareness and use of air quality indices, also examined broad levels of public concern about air quality and pollution levels. The study found that Canadians are aware of the issue of air quality and know it has the potential to substantially affect health. Air pollution is the dominant top-of-mind health hazard (mentioned by 33 per cent of respondents as their first response). Nearly half, 46 per cent, mentioned it among their top three responses. Nine in ten believe that air pollution affects health at least somewhat.^{Footnote #edN15}

Also in 2010, POR^{Footnote 11} with First Nations on-reserve and Inuit living in the North was undertaken to assess their knowledge about, perceptions of, and behaviours related to environmental health. The results reveal that many First Nations on-reserve (seven in ten) and Inuit (six in ten) believe that environmental issues can be a source of health problems. Mould, water pollution and air pollution are the most commonly mentioned areas of concern.^{Footnote #edN16}

Follow up POR, conducted in 2013 with First Nations on reserve and Inuit living in the North revealed that general awareness of environmental issues has remained the same. More First Nations on-reserve (49 per cent) and Inuit (51 per cent) indicated that they were confident they could definitely take steps to protect their health, compared to 2010 (38 per cent and 31 per cent, respectively) when larger proportions said that they could probably take steps. Mould is still a top concern (53 per cent of First Nations say it is a major risk to their health, as do 40 per cent of Inuit) while, water pollution and air pollution also continue to be commonly mentioned areas of concerns.

3.1.3 Support for the continued need to build the evidence base on the link between environmental contaminants and health

Finding 3. The risks of exposure and the potential health effects of many substances are not fully understood. There is a need to continue to build the evidence base on the connections between environmental contaminants and health as the science continues to evolve.

A review of documentation and literature revealed that while risks posed by some environmental contaminants such as lead and mercury are well established, the science on the risks associated with these substances continues to evolve. For example, HC has recently assessed the most current science on the effects of lead, which has shown evidence of negative health effects occurring at even low levels of exposure to lead.^{Footnote #edN17} As well, exposure levels and the health effects of other environmental contaminants are less clearly understood. A recent scoping review on the association between early life environmental exposures and later development of common chronic diseases notes that the literature on determinants of health has often overlooked environmental contaminants (with factors such as healthy eating, physical activity and tobacco having taken precedence).^{Footnote #edN18}

Support for enhanced monitoring and surveillance, including biomonitoring, to build the evidence base on the link between exposure to environmental contaminants and health has been articulated at the federal level in several studies/reports. The Statistics Canada CHMS evaluation notes that in Canada, a number of program and policy initiatives that occurred from 2001 to 2006 identified a need for surveillance and monitoring of public health indicators, thus providing a direct or indirect impetus for the creation and ongoing support of a direct health measures survey. Table 5 provides examples of such initiatives.

Stakeholder groups, including environmental and health non-governmental organizations and health research groups, have also advocated for improvements to measurements and studies to address gaps in knowledge about the connections between environmental contaminants and health. These recommendations include, for example:

• enhancements to biomonitoring studies to address geographic gaps and to further explore sources of exposures, causal links between contaminants and specific health impacts, and the effects of interventions such as regulatory measures or other public health activities on an individual's exposure to contamination (National Collaborating Centre for Environmental Health, 2008);^{Footnote #edN20}
• additional First Nations-oriented research for all topics in environmental health to provide baseline environmental monitoring and health data, link health complaints and environmental impacts, support professional development within the environmental health field and promote communities of practice among professionals (AFN, 2008);^{Footnote #edN21}
• calls for Canada to monitor toxic substances in urine, blood, and other human tissues to feed into a national health tracking system and to inform the public and health professionals, and to hold industry accountable for the release of toxic substances. Recommended actions included comprehensive data tracking (including children under the age of six), a national study on the overall burden of environmental hazards in Canada, and development and publication of a list of environmental health indicators (David Suzuki Foundation, 2007).^{Footnote #edN22}

Key informants across all respondent groups agree that there is a compelling and continued need to build the evidence base on the links between environmental contaminants and health. Key informants point to previous gaps in the Canadian evidence on exposures to environmental contaminants, particularly within vulnerable populations (e.g., First Nations on-reserve, children), and the importance of trend data (collected over the long-term using multiple data collection cycles) to determine the need for and effectiveness of risk management of harmful substances. Some key informants noted the ongoing need to assess and, if necessary, risk manage the multitude of chemical substances in use in Canada, only a small proportion of which has been examined in monitoring and surveillance studies to date.

3.1.4 Gaps in the evidence base

Finding 4. Action Plan funding addresses gaps in the evidence base through monitoring and surveillance activities.

The evaluation evidence indicates that Action Plan funding for monitoring and surveillance activities responded to identified gaps in health information in Canada. The CHMS evaluation, for instance, notes that CHMS was developed to fulfill specific health information needs, such as the need for:

• national baseline data on the extent of major health concerns by direct measures (e.g., obesity, hypertension, vitamin and nutrition deficiencies, chronic and infectious diseases, oral health status of Canadians and the level of access to dental care)^{Footnote 12};
• national data on exposure and prevalence levels of environmental chemicals in the population; and
• children's health information by extending the survey to children between the ages of three and six.

Similarly, the FNBI addressed a gap in health information in Canada with respect to adult First Nations living on-reserve (south of 60). First Nations may have the potential for greater exposure to some contaminants and the FNBI data, compared to nationally representative CHMS data, will help to identify substances to which First Nations (and sub-groups) may have more or less exposure.

With respect to the PHAC surveillance systems, the evaluation found that Canada's surveillance of congenital anomalies is limited although improvements have occurred over the past few years. Provincial and national perinatal surveillance systems do not include measures of environmental contaminants, and tracking of congenital anomalies is variable across jurisdictions and often incomplete (current records do not capture congenital anomalies diagnosed in infants over one month old or prenatal terminations). There is a lack of monitoring and surveillance of developmental disorders such as ASD, although some data exist outside the health domain (i.e., the education system).

3.2 Link to Government Priorities and Departmental Strategic Outcomes

3.2.1 Concurrence of Action Plan objectives and federal government health and environmental priorities and agenda

Finding 5. The Action Plan is consistent with federal government priorities and contributes to addressing the federal strategic outcome: Healthy Canadians. The Action Plan further supports broader federal health and environmental efforts, such as the Chemicals Management Plan and the Clean Air Agenda, as well as Canada's international obligations.

Concurrence with federal government priorities

Within the Whole of Government framework^{Footnote 13}, one of four key spending areas is Social Affairs, which includes the outcome area "Healthy Canadians". The Action Plan's long-term outcome to "reduce health risks to Canadians and particularly to vulnerable populations from environmental contaminants" falls within this outcome area.

Other documentary evidence indicates that the Action Plan is well-aligned with recent and current federal government priorities. Among the components funded by the Action Plan, the CHMS was an ongoing activity; the initial funding for the CHMS survey was outlined in the 2003 Federal Budget, where the Government of Canada highlighted the importance of "the availability of accurate and timely information on trends in health status and health system performance as a crucial tool to inform responsive, patient-centered health policy decisions." ^{Footnote #edN23} In the Speech from the Throne 2007, Improving the Environment and Health of Canadians was identified as one of the Government's core priorities: "Threats to our environment are a clear and present danger that now confronts governments around the world. This is nowhere more evident than in the growing challenge of climate change. Environmental protection is not just about protecting nature. It is about the health of Canadians. Recent events have called into question the safety of basic products such as food for our families and toys for our children." ^{Footnote #edN24} Budget 2008 provided funding for the Action Plan "to help Canadians make environmentally healthy decisions… and [for monitoring and surveillance focusing on] the relationship between environmental contaminants and congenital anomalies and childhood developmental disorders, and on examining the links between environmental contaminants and the risk of illness." ^{Footnote #edN25}

More recently, in 2010, Canada's Health and Health Promotion/Healthy Living Ministers endorsed Creating a Healthier Canada: Making Prevention a Priority: A Declaration on Prevention and Promotion from Canada's Ministers of Health and Health Promotion/Healthy Living. The Action Plan appears consistent with the guiding principles of the Declaration that health be addressed through a collaborative approach to promote health and prevent disease, disability and injury. The Action Plan components are also aligned with the Guiding Principle of Health promotion which includes, among other strategies, "Helping people learn and practise healthy ways of living" [EHGs], and "Doing and using research to build the evidence on what creates good health, the broad causes of disease and injury and how to influence them [monitoring and surveillance]". ^{Footnote #edN26}

Concurrence with federal health and environmental agenda

Prior to the funding of the Action Plan, two major federal government initiatives were underway, the 2006 Chemicals Management Plan and the 2007 Clean Air Agenda, to address issues related to environmental contaminants. The environmental health guides and monitoring and surveillance activities of the Action Plan were intended to support these initiatives by expanding the reach of information available to Canadians under the CMP and CAA (through Hazardcheck and the First Nations and Inuit EHGs), as well as by providing information to decision-makers on the effectiveness of current policies, the need for new interventions, and priorities for action and for research. ^{Footnote #edN27}

Documentary evidence and key informant interview findings confirm that the Action Plan has supported the CMP and CAA initiatives. The links between the Action Plan and the CMP are particularly close. The Action Plan's intended ultimate outcome of "reducing health risks to Canadians (particularly vulnerable populations) from environmental contaminants" aligns with the intermediate outcome of the CMP that "risk to Canadians' health and their environment posed by harmful chemicals is reduced".

With respect to public awareness, the Hazardcheck guide developed under the Action Plan currently resides within the Program Development and Stakeholder Engagement Division of the HECSB, responsible for public outreach and stakeholder engagement under the CMP. Hazardcheck is identified as a tool within the Division's 2012-2013 Environmental Health and CMP General Population Outreach -- A National Approach strategy document to provide Canadians with information about indoor environmental health hazards and actions to reduce their exposure. Most key informants agree that Hazardcheck supports the public outreach component of the CMP by helping to "put a public face" on the CMP. Public outreach and proactive communications to Canadians was noted as a shortcoming in the 2011 evaluation of the CMP. A few key informants note that while the publication has a close tie to the safe use of chemicals, Hazardcheck is less relevant to communicating the regulatory aspects of the CMP and focuses on a very select number of chemicals (see Appendix D for a listing of Hazardcheck and First Nations and Inuit EHG topics).

Concerning monitoring and surveillance, the Action Plan supports the CMP through funding of two key surveys: the CHMS infrastructure and collection of data from Canadians ages 3 to 79; and the FNBI which collects data from First Nations people living on reserve with respect to exposure to chemical contaminants. The CMP, itself, also funds the biomonitoring component of the CHMS. The CHMS and FNBI data, together with the surveys conducted under the Northern Contaminants Program^{Footnote 14} (NCP) (North of 60) and Maternal-Infant Research on Environmental Chemicals Study^{Footnote 15} (MIREC), are the basis for establishing concentrations of chemicals in people, providing trend data, and contributing to the understanding of the relationships between chemicals, nutrition and chronic disease. These data allow the formulation of research questions or hypotheses about the causal nature of relationships that can be investigated through further study. The levels of concentrations of chemicals (above or below reference levels) and the proportion of the population above reference levels may signal a need for a risk management strategy. Ultimately, monitoring and surveillance activities, when mature (i.e., over multiple cycles), will assist in determining the effectiveness of risk management (e.g., regulatory or public health interventions).

The links between the Action Plan and the CAA are considerable, though perhaps not as clearly articulated as the CMP. Initiated in 2007-2008 and renewed in 2011-2012, the CAA included themes related to Indoor Air Quality (IAQ) and Adaptation. The Action Plan supports the CAA by addressing the information needs of Canadians in the area of indoor air quality, including raising public awareness about the linkages between indoor air quality and potential health effects. Topics within Hazardcheck and the First Nations and Inuit EHGs address radon and radon testing at home, as well as other indoor air quality issues (e.g., carbon monoxide, second-hand smoke). The CHMS and FNBI suite of chemicals address some indoor air components to assess exposures to these chemicals found in the home (see Appendix E for a listing of chemicals monitored by the CHMS and FNBI).

International commitments

The monitoring and surveillance activities funded by the Action Plan, particularly the CHMS, support Canada's commitments internationally. The CHMS evaluation, for example, noted that survey information helps to address the federal government's international commitments and obligations, such as the North American Free Trade Agreement, the Canada-U.S. Great Lakes Water Quality Agreement, United Nations negotiations on the Global Treaty for Mercury, and the Stockholm Convention on Persistent Organic Pollutants. The FNBI supports the Global Treaty for Mercury and the Stockholm Convention which specifically require monitoring of populations that are vulnerable to exposure. In addition, enhanced congenital anomalies surveillance and the surveillance of developmental disorders would also contribute to the global efforts in improving child health, as pledged by the federal government at the Muskoka Initiative (G8 Summit in June 2010) and the Commission on Information and Accountability for Women's and Children's Health (as relevant to Millennium Development Goal 4).

3.2.2 Alignment with departmental strategic outcomes and priorities of the three partners

Finding 6. The funded components of the Action Plan are aligned with partner departments' strategic outcomes and priorities.

The following table summarizes the links between the component programs of the Action Plan and partner departments' strategic outcomes and Program Alignment Architecture (PAA).

The Action Plan component programs are consistent with priorities identified in departments' respective Reports on Plans and Priorities (RPP), as follows.

Health Canada

The HC RPPs in 2011-2012 and 2012-2013 state that ensuring that Canadians have credible information on the impact of chemicals in the environment and the steps that they should take as a result is a priority to address their concerns about environmental contaminants. ^{Footnote #edN28} Mechanisms such as monitoring and surveillance and research are required in order for HC, PHAC and others to be able to address environmental contaminants and their potential effects on human health. The data and research are also necessary for assessing the effectiveness of interventions to mitigate risks and identifying emerging risks to human health from environmental contaminants. The most recent RPP further indicates that in 2012-2013 HC aims to promote and protect the health of Canadians by identifying, assessing and managing health risks posed by environmental factors, including research, risk assessment and management of chemical substances, air pollutants, and water contaminants.

Statistics Canada

Supporting CHMS and other projects that provide statistical information and analysis about the state of Canadians' health is set out as one of Statistics Canada's activities in the 2011-2012 RPP.

PHAC

In the 2008-2009 RPP, PHAC committed to elaborating a comprehensive policy on possible health effects caused by the physical environment by partnering with different federal and other organizational partners. In this report, the Agency recognizes that "Changes in climate, air and water quality, wildlife habitats and other aspects of the environment all have an impact on the health of Canadians."^{Footnote #edN29} As well, by working with Health Canada's Healthy Environments and Consumer Safety Branch, the Public Health Agency continues to "[address] the links between health and the environment."^{Footnote #edN30}

3.3 Appropriateness of the Federal Government's Role and Responsibility

Finding 7. The federal government's role in the Action Plan as a whole, and at the component level, is appropriate.

3.3.1 Federal Role

Documentary evidence indicates that the Action Plan supports the federal government in fulfilling its legislative obligations in the area of environmental health. Based on the documents reviewed for the PHAC surveillance systems evaluation data collection exercise, the document concludes that while the promotion and preservation of the health of Canadians is a shared jurisdiction between the provincial/territorial and federal governments,^{Footnote #edN31} the federal government has authority, for example, to collect, analyse and disseminate public health data.^{Footnote #edN32} Monitoring and surveillance data inform and support the federal government in its ongoing monitoring, surveillance, risk management and regulating activities, which are set out in legislation such as the Department of Health Act, the Canadian Environmental Protection Act, the Pest Control Products Act, the Food and Drugs Act, and the Canada Consumer Product Safety Act.

The Action Plan components led by the FNIHB -- the First Nations and Inuit EHGs and the FNBI -- are also consistent with the federal government's responsibilities with respect to the health of First Nations and Inuit with the federal government supporting the delivery of health care in First Nation communities (on-reserve) and Inuit communities. While these roles and responsibilities are evolving, HC through FNIHB continues to support the funding or provision of primary care services on-reserve in remote and isolated areas, and public health, health promotion and environmental health services on reserve and in Inuit communities within its stated roles. Under the Department of Health Act, the Minister of Health provides First Nations and Inuit health programs and services. There is no specific legislative foundation for these programs and services which are approved annually by means of expenditures proposed in the Estimates and authorized by an Appropriations Act. These programs and services are provided or funded by FNIHB in a manner consistent with the 1979 Indian Health Policy, departmental policies and mandates. The Canadian Environmental Protection Act (CEPA, s.45) authorizes the Minister of Health to conduct research and studies relating to the role of substances in illnesses or in health. The HC PAA for Strategic Outcome 3 (First Nations and Inuit communities and individuals receive health services and benefits that are responsive to their needs so as to improve their health status) also articulates HC's role and responsibilities with respect to health in First Nations communities. Primary health care includes health promotion and disease prevention, public health protection (including surveillance), and primary care. FNIHB's Strategic Plan 2012 highlights FNIHB's programs and services which include programs to control communicable diseases and address environmental health issues in First Nations and Inuit communities outside the Territories, and initiatives related to environmental health risk assessment and contamination.^{Footnote #edN33}

Key informant interview findings indicate a widespread perception that there is an important role for the federal government to provide credible and nationally representative survey measures for comparability, consistency, and objectivity. Key informants noted that national monitoring and surveillance data support the federal (HC) mandate to protect the health of Canadians, as well as the federal regulatory role and responsibility for CEPA toxic substances. According to key informants, the federal level has credibility with the general public to address issues related to public health (including both research and public awareness), and has the mandate, resources, and scientific expertise to undertake activities of a national scope. Key informants point out that, internationally, similar programs are also delivered at a federal level (described in more detail in Section 3.3.3).

3.3.2 Departmental Mandates

The reviews of documentation conducted by each department indicate that the three Action Plan partner departments are leading components of the Action Plan that are consistent with each one's respective mandate. These findings are summarized below.

Health Canada

The Department of Health Act extends authority to the Minister of Health over matters pertaining to the promotion or preservation of the health of Canadians that are within the federal jurisdiction. According to HC's mission and vision, "HC is the federal department responsible for helping the people of Canada maintain and improve their health. HC is committed to improving the lives of all of Canada's people and to making this country's population among the healthiest in the world as measured by longevity, lifestyle and effective use of the public health care system."^{Footnote #edN34}

HC plays five core roles^{Footnote #edN35} in the area of health. Notably, to fulfill its stated roles, the Department draws on strengths as a science-based department. One of five stated roles for HC is Information Provider: "performing high quality science and research, we support policy development, regulate increasingly-sophisticated products and provide the services, information and management essential to affordable and world-class health care for Canadians. Through research and surveillance, we provide information that Canadians can use to maintain and improve their health".^{Footnote #edN36} Components of the Action Plan, including the EHGs and FNBI, are consistent with this role through contributions to building the evidence base on exposure to environmental contaminants and linkages with health, and dissemination of information on environment health risks and actions to mitigate risks to Canadians.

Statistics Canada

The Constitution Act, 1867 establishes "census and statistics" as an area of federal jurisdiction. Parliament has exercised its responsibility for the census and statistics primarily through the Statistics Act. The Act creates Statistics Canada as Canada's national statistical office and establishes its mandate, powers and obligations. Under the Act, Statistics Canada must collect, compile, analyze and publish statistical information on the economic, social and general conditions of the country and its people. Under section 22(c) of the Statistics Act, Statistics Canada has the mandate to collect, compile, analyze, abstract and publish statistics on health and welfare.

PHAC

The Public Health Agency was established in 2004 by the Public Health Agency of Canada Act^{Footnote #edN37} to promote and protect the health of all Canadians by showing leadership in federal efforts, partnership, innovation and action in the field of public health.^{Footnote #edN38} The Public Health Agency also has a role to assist the Minister of Health presiding over the Agency ^{Footnote #edN39} in collecting, analysing, interpreting and disseminating information pertaining to public health.^{Footnote #edN40}

One of the Public Health Agency's core functions is public health surveillance,^{Footnote #edN41} including enhancing the quality and quantity of surveillance data and expanding the knowledge of disease and injury in Canada.^{Footnote #edN42} Core surveillance areas include maternal and child health surveillance systems which encompass data on injury, abuse, perinatal diseases and new developmental disorders, as well as adult chronic disease surveillance of cancer, arthritis, diabetes, respiratory, mental illness, cardiovascular and risk factors.^{Footnote #edN43} The Public Health Agency's mandate includes the support of "effective public health action within PHAC and among its partners."^{Footnote #edN44}

3.3.3 International Context

Finding 8. Federal delivery of the key activity areas under the Action Plan is consistent with the approaches of other jurisdictions internationally.

According to documentation and literature reviewed for the CHMS evaluation and the roll-up evaluation, several countries have a history of conducting surveys that include direct health measures, including biomonitoring that have yielded important findings, validating the need for this type of survey. For example, the U.S. National Health and Nutrition Examination Survey (NHANES),^{Footnote #edN45} conducted since the early 1960s,^{Footnote 16} provided the first evidence that Americans had high lead levels in their blood, which motivated governments to phase out the use of lead as an additive in gasoline and paint.^{Footnote #edN46} In Australia, a national direct health measures survey conducted from 1999 to 2001 found that for every known case of diabetes, there was one undiagnosed case, and that nearly one million Australians over age 25 have diabetes.^{Footnote #edN47} Finland, too, has a legacy of important public health and scientific findings from national direct health measures surveys.^{Footnote #edN48},^{Footnote #edN49},^{Footnote #edN50}

Surveillance of health issues such as autism spectrum disorder or other developmental disorders is challenging, and this is substantiated in the international literature. The World Health Organization, in their latest classification of mental and behaviour disorders, state that there is uncertainty regarding the precise definitions of developmental disorders such as autism, even though there is general agreement on the overall conceptualization of these disorders.^{Footnote #edN51} These methodological issues, including the variability in determining autism spectrum disorder prevalence, may impede international benchmarking in this area (although estimates of autism spectrum disorder no longer differ widely between countries in the Americas, Western Pacific and Europe)^{Footnote #edN52}.

Literature was reviewed in the roll-up evaluation pertaining to international examples of health and environmental action plans. This review indicates a rich experience in the European Union (EU) in national action plans to address environmental health. Experience in these jurisdictions and others such as the US and Australia suggest a number of common features of these action plans. Like Canada, they are carried out on a national level, include a focus on vulnerable populations (especially children), and give attention to scientific inquiry (including biomonitoring) to build the evidence base on the links between environmental exposures and health, and to assess the effectiveness of policy to mitigate negative health effects.

3.3.4 Overlap and Duplication

Finding 9. While there are other environmental health public awareness and monitoring and surveillance activities evident at the federal level and in other jurisdictions in Canada, there were few concerns that duplication has undermined the role of the federal contribution in these areas.

Public awareness

The documentation review and key informant interview findings suggest that there is an array of information available in the public domain on topics pertaining to environmental health. Sources include HC (separate strategies on topics such as radon and mould), other federal departments (Canada Mortgage and Housing Corporation material on healthy homes), provincial/territorial governments and non-governmental organizations (including a number of Aboriginal health organizations). However, the evaluation found that available materials more often focus on specific topics, especially topics such as air pollution, lead, mercury, carbon monoxide, and mould. This was confirmed by many key informants who characterized the Hazardcheck and First Nations and Inuit EHGs as more 'holistic' in their approach. A few key informants noted that Hazardcheck and the First Nations and Inuit EHGs identify the high priority issues that Canadians should pay attention to, distill the information that is available and scientifically credible, and provide practical advice in a concise manner in a single publication.

According to key informants, the Action Plan funding for environmental health guides and monitoring and surveillance has been very important. Most key informants are of the opinion that without Action Plan funding, Hazardcheck and the First Nations and Inuit EHGs would not have been developed in their present form, nor would the marketing campaigns have been as extensive. However, according to a few key informants, information pertaining to individual environmental health topics would have been available in a more piecemeal fashion through other sources, including the government's own strategies pertaining to issues such as radon.

Monitoring and surveillance

Evaluation findings based on documents and key informant interviews indicate that there is no duplication between the CHMS and FNBI Action Plan components and monitoring and surveillance activities within other jurisdictions. The CHMS and FNBI uniquely collect nationally representative direct health measures survey data for Canadians overall (CHMS) and First Nations on-reserve (FNBI). Similarly, while some provinces/territories and other organizations collect surveillance data on congenital anomalies and developmental disorders through local or provincial databases and registries, the PHAC surveillance systems are anticipated to bring together and enhance these existing systems to generate reliable and valid national level data.

Other examples of health monitoring and surveillance efforts are carried out at the provincial/territorial and community levels. For example, both Saskatchewan and Alberta undertake biomonitoring surveys that include First Nations with a focus on monitoring pregnant women to examine levels of selected environmental contaminants. There are examples of community-level studies that have examined causal links between environmental contaminants and adverse health effects. This includes community-initiated research funded by the FNIHB First Nations Environmental Contaminants Program. However, while valuable and complementary, these data collection efforts do not generate national-level data and are based on methods not used by the CHMS and FNBI (e.g., they use self-selected sampling, do not include physical measures, or use pooled serum sample results). Another example of monitoring and surveillance data collection which is complementary to the FNBI is the First Nations Food, Nutrition and Environment Study which collects regionally representative data, and focuses on diets and food-related exposures to contaminants in traditional foods and drinking water consumed by First Nations. Finally, there are international monitoring and surveillance data. These data can be (and have been) used as proxy measures or to compare and validate Canadian data, but are not ideally suited for Canadian regulatory and policy-making due to differences in Canadian climate, demographics and geography which can influence exposure levels and health effects.

According to key informants, the Action Plan filled a critical gap in funding the infrastructure for the CHMS, and enhancing the robustness of monitoring and surveillance data through the extension of the survey to children and to cover First Nations on-reserve through the FNBI. According to the CHMS evaluation findings and confirmed in the roll-up evaluation, had subsequent cycles of the CHMS not been funded by the Action Plan, this would have represented a significant loss of investment in establishing the CHMS infrastructure and would have had impacts on researchers' and policy makers' understanding of health in Canada, especially for children aged 3 to 11. Other impacts that were noted in the CHMS evaluation included loss of the ability to adjust self-reported data based on objective direct measures and, in the health policy area, impaired decision making based on exposure levels and outcomes of regulatory or public health interventions.

4 Findings - Performance

This section presents the evaluation findings related to the achievement of the Action Plan's intended outcomes. At the level of intermediate outcomes, the achievement of outcomes related to the environmental health guides is assessed separately from the intended outcomes that are common to the monitoring and surveillance components of the Action Plan. Progress toward achievement of the ultimate outcome of the Action Plan is examined, although this outcome was not anticipated to be achieved within the period covered by this evaluation.

4.1 Intended Immediate Outcomes

4.1.1 Increase in public awareness of connections between environmental exposures and health

Finding 10. There is evidence that Canadians are aware of connections between environmental exposures and health; changes in awareness differ across target groups. Statistics on publication distribution indicate that outreach performance targets were exceeded in this area for Hazardcheck and First Nations and Inuit environmental health guides.

The performance target for the public awareness component of the Action Plan was to increase awareness among Canadians by five per cent over the five years of the Action Plan. However, the document review revealed a limited amount of POR conducted on awareness of the connections between environmental contaminants and health, and this was confirmed by key informants. To date, the only relevant Canada-wide POR available was conducted in 2010 as part of the campaign/communications evaluation^{Footnote 17} of the Hazardcheck advertising campaign.^{Footnote #edN53} Although the research did not question 'awareness' explicitly, it did include questions that can be used as a proxy for awareness. The research found that 55 per cent of Canadians believed that their health was affected by environmental problems (substantially higher than the proportion -- three in ten reported above -- that believe their health has been adversely affected by chemicals). This represents a decrease from 2007 when, responding to a comparable question, 63 per cent of Canadians indicated a connection between environmental problems and health (reported and cited earlier in Section 3.1.2). In the 2010 research, the majority of Canadians believed that their health had already been affected by air pollution (59 per cent), pesticides in food (52 per cent), or chemical pollution (51 per cent). Nearly half (46 per cent) thought that second hand smoke from tobacco had already affected their health, while approximately one-third thought this to be the case regarding climate change (36 per cent) and mould (32 per cent). When asked which environmental problem or hazard poses the greatest risk to the health of Canadians, air pollution/smog (41 per cent) followed by water pollution (19 per cent) were Canadians' top concerns among potential environmental hazards. One in ten each identified pollution (generally), global warming/climate change, toxic chemicals, and food safety/contaminated food as a health risk.

While not a measure of awareness, distribution and outreach statistics summarized in marketing reports conducted for Hazardcheck provide an indication of the reach of the campaign. These data indicate that distribution of the Hazardcheck was well in excess of the performance target of 10,000 guides distributed per year. In total, about 855,000 print copies of Hazardcheck were distributed between its initial publication in March 2010 and March 2013. A marketing campaign for Hazardcheck initiated in March 2010 using print (magazine and newspaper), radio and digital advertising (web banner, Google Adwords buy) and out-of-home^{Footnote 18} advertising (digital billboards) appeared to yield significant initial traffic to the Hazardcheck website. About 85,000 visits to the website were recorded during this month. This generated, in turn, a spike in online requests for the publication from the HC publication distribution centre.

Further social marketing activities included, in 2010-2011, retail events in home renovation stores across Canada, a Google Adwords buy, enhanced website content, and polybagging of the Hazardcheck guide in the Parents Canada magazine. The in-store events resulted in 15,000 one-on-one conversations with consumers and distribution of 124,000 guides, and the Parents Canada circulation totalled 180,000 guides.

The following year saw renewed interest in retail events, a Google Adwords buy and print polybagging, in addition to social media efforts. The in-store events led to 30,000 one-on-one conversations and the distribution of 9,900 guides and 36,000 checklists; 180,000 guides were distributed via Parents Canada; and the blogger outreach had a 10% blogger uptake.

The last year of the campaign (2012-2013) built on the success of the retail events by involving a more diverse set of retailers, resulting in 34,000 one-on-one conversations, the distribution of 100,000 Hazardcheck guides, and promotion of the events and the campaign to 39,000 subscribers of like-minded organizations.

Regions and Programs Bureau (RAPB) staff who work in partnership with the Program Development and Stakeholder Engagement Division of the HECSB (responsible for public outreach under the CMP) also play a role in the distribution of Hazardcheck through their regional networks and events. In 2011-2012, RAPB participated in and/or implemented approximately 78 outreach activities/events. These included home and garden shows, health fairs, Eco Fests, health professional conferences, industry trade shows, capacity building workshops, and Children's Environmental Health Sessions.^{Footnote #edN54} The Regions reported distributing a combined total of approximately of 17,000 copies of Hazardcheck and 900 copies of the Hazardcheck Checklist.

Campaign tactics differed between Hazardcheck and the First Nations and Inuit EHGs, based on research on best practices for health promotion for First Nations and Inuit populations and advice provided by partners and an advisory group. Like the Hazardcheck, distribution channels included direct mail and online ordering. However, the distribution of the First Nations and Inuit guides also focused on community-based distribution through community organizations (including schools) and leveraged local expertise -- HC Environmental Health Officers (EHOs) and Regional Environmental Health Managers (REHMs) -- for distribution. In addition, the First Nations and Inuit EHGs have had a distinct focus, including unique products and distribution strategies, for youth. Distributions of youth guides and companion materials in schools, as well as complementary strategies such as art contests, have been used for these populations. Considering all the versions of the EHGs and companion documents together, about 425,000 guides for First Nations and 67,300 for Inuit were distributed (as of March 2013). An internal survey (n=45) conducted with HC's EHOs and REHMs indicates that the campaigns for the First Nations and Inuit EHGs have reached communities, and products are being disseminated.^{Footnote #edN55} For example:

• Over eight in ten EHOs (84 per cent) say they have received the First Nations Home Guide and 62 per cent received the Spring & Summer Outdoor Guide. Awareness of some of the companion materials and targeted products (e.g., for children and teens) and online resources was lower.
• Three-quarters of EHOs/EHMs who have received the campaign resources (77 per cent) say they distributed the resources at health centres, followed by distribution to clients during home inspections or visits (55 per cent) and via the Band office (55 per cent).
• Most EHOs/EHMs indicated that the Environmental Health campaign is very or somewhat useful at: increasing the knowledge of simple activities that can be done to reduce and prevent harm to health (78 per cent); increasing the awareness of the relationship between environmental contaminants and health (73 per cent); and encouraging community members to seek out information on environmental health (65 per cent).

A post-campaign survey measuring awareness of environmental health issues among First Nations and Inuit was conducted in March 2013, with results compared to baseline data collected in 2010.^{Footnote 19},^{Footnote #edN56} Findings included:

• A majority of First Nations (79 per cent) and Inuit (51 per cent) said that environmental issues affect their and their family's health a fair amount or a great deal.
• The proportion of First Nations on reserves feeling the environment affects their own or their family's health "a great deal" grew from 2010 (52 per cent, up from 42 per cent) but fewer Inuit said the same in 2013 (22 per cent, down from 36 per cent).
• Larger proportions of First Nations and Inuit were able to identify, without prompting, some form of environmental issues that affect health (84 per cent of First Nations versus 69 per cent in 2010, and 72 per cent of Inuit versus 57 per cent in 2010).
• Overall awareness of the marketing campaign was quite high: four in ten First Nations (39 per cent) and Inuit (40 per cent) recalled at least one of the First Nations/Inuit EHG materials.

Often pointing to distribution statistics and anecdotal evidence, some key informant interviewees were of the opinion that there has been an increase in public awareness as a result of Hazardcheck and the EHGs for First Nations and Inuit. They noted that the guides and their associated products have been promoted widely, using a variety of distribution channels. A review of other environmental health-related campaigns confirms that the marketing of the guides appears to have been quite broad-based in its use of conventional and innovative marketing tactics compared to efforts in these other jurisdictions.

Documentary evidence and the views of key informants give particular importance to the use of online activities to promote the guides and to drive traffic to the website to view the guide, companion documents, and associated interactive tools online. For example, POR conducted to evaluate Hazardcheck found that the top source for information on environmental health issues identified by Canadians is the Internet/websites (64 per cent). This finding was used to adjust the Hazardcheck campaign by focusing on online advertising efforts (e.g., by continuing with Google advertising, reaching out to bloggers, and sharing more content through social media), and minimizing the use of radio and print advertising. The campaign also evolved to target the advertising more precisely to audiences who are most receptive and most at risk (e.g., parents, new home buyers, those with chronic illness). In follow-up POR with First Nations and Inuit, one-quarter of Inuit (25 per cent) recalled the Facebook page created for the campaign; recall among First Nations was lower (12 per cent).

While CHMS and FNBI do not formally link to intended outcomes pertaining to public awareness, several key informants noted that the CHMS and FNBI have raised awareness of the connections between environmental contaminants and health among the participants in these studies. Preparatory communications with participants and with the selected communities for the FNBI, as well as communication of individual and community-level laboratory results, have reportedly led to an increased understanding of exposures and sources of exposures to environmental contaminants. A small number of representatives from the participating communities who were interviewed requested more information and guidance in interpreting the results (e.g., results presented for non-technical audiences, cross-community comparisons/lessons learned, a more multi-faceted strategy that would include an education component based on biomonitoring results, guidance on stewardship of the data). Additionally, while not directed to a general public audience, results of the CHMS and FNBI will be posted publicly and published in scientific journals and at conferences.

4.1.2 Availability of reliable and usable data to enable comparisons between First Nations and the Canadian population

Reliability and Usefulness of FNBI data

Finding 11. FNBI has contributed to reliable data that are comparable to CHMS data at the population level (20 years and above). Comparability is, however, limited at the level of subpopulations (e.g. age and region) due to FNBI sample size limitations. It is too early to assess usefulness as data have only recently been publicly released.

According to program documentation, and confirmed by key informants, the FNBI was designed as a direct health measures survey, which replicated the CHMS methodology in terms of selection of measures (the FNBI monitors all of the chemicals included on the CHMS (Cycle 1)) and analysis. The FNBI includes biomonitoring (collection of blood and urine samples), direct physical measurements, and a household questionnaire. Over 500 individuals from 13 First Nations communities participated in the study to produce data that are representative of the First Nations on-reserve population in Canada.^{Footnote 20} A review of the FNBI method and sampling conducted by Statistics Canada indicated that the survey should provide estimates comparable to CHMS and that while some statistical issues were encountered during implementation, they would not have a large impact on the usefulness of the results.^{Footnote #edN57} According to key informants, the FNBI collected new data on the health status of First Nations' people living on reserve, providing reliable estimates comparable to CHMS estimates for the Canadian population aged 20 and above.

Key informants noted that the comparability between the FNBI and CHMS is further enhanced through the use of the same laboratories for analysis of the biological samples, as well as the same transportation protocols. According to interview findings from the roll-up evaluation, Statistics Canada worked in consultation with AFN and FNIHB to ensure comparability of the analysis to the CHMS data. The CHMS evaluation similarly noted that the program worked in collaboration with HC's FNIHB on the FNBI. Both the CHMS and roll-up evaluation identified that using comparable methods, procedures, tools, and reference labs created efficiencies for FNIHB at the operational level, but also provided better comparability of the results because the same measures were used in First Nations population as in the general Canadian population. Room for even more collaboration between the two initiatives was noted in the evaluation of the CHMS.

FNBI methodological documentation discusses the primary limitation of the comparability of the FNBI data with CHMS. While the FNBI was designed to provide comparisons between First Nations and the Canadian population, the FNBI was not funded to obtain a sufficient sample size (n=502) to allow for multiple age groupings in the results analysis, therefore the decision was made that within the FNBI, only one age grouping would be used (aged 20 and above). By comparison, within the CHMS (n=5,600 and 5,700), sampling occurred among pre-determined age groups (6 to 11, 12 to 19, 20 to 39, 40 to 59, and 60 to 79). These age categories are now further expanded in the CMHS as the Action Plan funding has extended data collection to include children under the age of six.

With respect to usefulness, the FNBI national data were only recently made public so the perceptions of federal or external data users could not be assessed. Community reports were prepared for each of the 13 participating reserves and presented to these communities, and a national report^{Footnote #edN58} was published in June 2013. Some interviewees indicated that the FNBI data are useful, although a small number suggested data could be made more useful to community members through increased guidance from FNIHB and AFN on interpretation. According to Health Canada representatives, this is a difficulty given the limitations of current scientific knowledge on the causal relationships between environmental chemicals and adverse health effects, but should be more feasible when additional data is made available following the next survey.

Reliability and Usefulness of CHMS data

Finding 12. Survey participation targets for the CHMS have been achieved. The CHMS has contributed to the availability of reliable and usable data to confirm levels of exposure to environmental contaminants, and internal and external researchers are satisfied with data quality. The CHMS data also serve to validate data collected from other sources.

According to the Statistics Canada evaluation of the CHMS, two cycles of data collection for the CHMS have been completed and a third is in progress. Targets of 5,600 participants for Cycle 1 and 5,700 participants for Cycle 2 were achieved. The planned number of participants for Cycle 3 is 5,700, which is comparable to that of the previous two cycles. The content of the CHMS addresses emerging environmental issues through: measures of chemicals in tap water (11 measures collected in Cycle 3); indoor air (102 measures collected in Cycle 2 and 81 measures in Cycle 3); and through biomonitoring (76 measures in Cycle 1, 101 measures in Cycle 2 and 67 measures in Cycle 3). Biomonitoring samples were taken from children under the age of 6 in both Cycles 2 and 3.

Based on survey administrative data, the CHMS evaluation identified that the response rates were high for both Cycle 1 and Cycle 2 with response rates of 88 per cent for the household questionnaire and 85 per cent for the mobile clinic for Cycle 1 and 90 per cent for the household questionnaire and 82 per cent for the mobile clinic for Cycle 2, which is an indicator of quality and reliability of the data. In designing the CHMS, Statistics Canada replicated many of the features of the US National Health and Nutrition Examination Survey (NHANES)--a survey that has existed for the last 50 years. Survey planning is extensive and content priorities are determined through a federal governance structure, in consultation with other stakeholders.

The CHMS evaluation notes that prior to the CHMS, Statistics Canada had been collecting data on the health status of Canadians and related issues through self-reported surveys. Direct health measurements, such as those collected by the CHMS, help validate self-reported surveys. These data are needed for public health education, health promotion programs, health care planning, health surveillance, and research.

Findings from the Statistics Canada evaluation of the CHMS found that most of the key informants interviewed feel that CHMS data are reliable and useful. Three quarters of data users who were surveyed for the CHMS evaluation also indicated that they were satisfied or very satisfied with the reliability and validity of the CHMS data. Findings from the key informant interviews conducted for the roll-up evaluation support the findings from the CHMS evaluation; monitoring and surveillance data users were generally positive with respect to the reliability and usability of the CHMS data.

A limitation of the CHMS data noted in the CHMS evaluation and in the roll-up evaluation is with respect to the lack of sufficient sample to conduct analyses at the sub-national level. Sample size is also a limitation for analysis on specific subsets of the population (e.g., vulnerable populations, people with rare conditions) and in the first cycle of the CHMS there was insufficient data on children (a situation that was rectified by Action Plan funding which extended the age of participants to include children three to six years of age). However, the Statistics Canada evaluation of the CHMS notes that the CHMS was intended to be a national survey and provision of data at a sub-national level is beyond the scope of the CHMS (although Cycle 1 and Cycle 2 CHMS data have been combined to permit sub-group analyses).

A final concern expressed by some key informants with respect to the CHMS (as well as the other monitoring and surveillance data components of the Action Plan) is the limitations of the data in establishing causal relationships between environmental contaminants and adverse health effects. This is not a deficiency of the components themselves, but rather a limitation of monitoring and surveillance data in general. The literature on direct health measures and human biomonitoring studies provides some indications of the general uses of these data (e.g., to determine concentrations of substances and identify where exposures are higher or lower within segments of the population, and to help to set an agenda for regulatory follow-up public health intervention), as well as their limitations. Bates et al., for example, note that while biomonitoring studies provide data that confirm exposure to chemicals at some point, limitations exist such as:

• Determining how the levels of environmental chemicals measured in an individual can be used to establish the level of environmental exposure that the individual experienced and the time-frame over which the exposure occurred;
• Identifying the background or baseline level (across the general population) of a given chemical in a tissue that individual levels can be compared to; and
• Establishing a link between the levels of a chemical measured in tissue or fluids and individual and/or population health. ^{Footnote #edN59}

4.1.3 Availability of reliable and usable data on congenital anomalies/ developmental disorders

Finding 13. Although preliminary activities have progressed, to date there has been little surveillance data collected or made available related to the enhanced congenital anomalies system. Surveillance data for developmental disorders has not yet been collected.

Based on findings from the PHAC surveillance systems, reliable and usable data on congenital anomalies and developmental disorders is limited as there has been slower than anticipated progress due to, for example, negotiations with partners and privacy concerns. At this time, neither system is at a point where this immediate outcome can be measured.

Activities undertaken to date for enhancing the surveillance system for congenital anomalies have focused on the development and negotiation of memorandums of agreement (MOAs) with provinces/territories which transfer funding to establish or enhance congenital anomalies surveillance systems in these jurisdictions. Since 2008, PHAC has initiated 8 MOAs with provinces/territories and is currently in contact with others to develop/enhance congenital anomalies surveillance in their respective jurisdictions. There are now seven active systems for population-based congenital anomalies surveillance across jurisdictions compared to two systems before the implementation of the Initiative. It is expected that participating jurisdictions will begin data sharing with PHAC sometime within the next two years. In addition, the national system is working on enhancing its data mining capability to address identified limitations (e.g., extended length of follow-up) and to include new P/T data, once they are made available to PHAC.

Analysis of existing surveillance data on congenital anomalies, including the influence of environmental contaminants, and dissemination of surveillance results is also occurring under this component. According to internal key informants, some enhanced surveillance information that has been developed has been used for national reporting (e.g. Perinatal Health Indicators for Canada 2011, Congenital Anomalies in Canada 2013). Finally, annual scientific meetings of the Canadian Congenital Anomalies Surveillance Network have been held, which provide a forum for knowledge exchanges and sharing of best practices.

Establishing the surveillance system for developmental disorders has included a focus on initial preparatory activities. ASD in children and youth have been the focus of the development of a surveillance network. PHAC has established internal capacity in this area through a new Developmental Disorders Surveillance Unit. Consultations and/or program development activities are ongoing with provincial/ territorial jurisdictions and with the assistance of an Expert Advisory Committee around: establishing conditions to be monitored (such as an ASD case definition and data sets); governance of an ASD surveillance network, capacity in provinces and territories to undertake surveillance through completion of environmental scans; initiation of pilot and feasibility studies in select provinces/territories; and establishing sentinel sites or population-based surveillance systems in select jurisdictions. The delays in the establishment of the ASD surveillance system mean that data collection is currently planned to begin by 2015. The program is now in its implementation phase.

4.2 Intended Intermediate Outcomes

4.2.1 Increased action by Canadians to minimize environmental health risks

Finding 14. There is some evidence that there has been an increase in action by Canadians to minimize environmental health risks; the extent to which changes in behaviour will remain stable over time and become more broad-based is unknown.

The national survey of Canadians conducted in 2010 ^{Footnote #edN60} to evaluate the Hazardcheck advertising campaign provides some data that indicate that Canadians are taking action to minimize environmental health risks. The survey showed modest levels of recall of the Hazardcheck campaign. Nearly one-quarter of Canadians (23 per cent) had noticed advertising about environmental hazards in their home. Overall, one in seven (17 per cent) said, when prompted, they recalled hearing some element of advertising related to the campaign (somewhat lower than benchmark recall levels of 23 per cent for other federal campaigns that, like this one, did not include a television component).

Of those who recalled seeing the ads, 18 per cent indicated that they did something as a result, of which more than four in ten (44 per cent) said they made home improvements or renovations. Approximately one in ten said they started to participate in recycling or composting programs (12 per cent) or made safer/healthier purchases (10 per cent). The proportion saying they took action is similar to or better than other HC campaigns that were looked at in this evaluation.

Almost nine in ten Canadians overall believe that steps can be taken to protect oneself from environmental health hazards, and the proportion who report having taken steps to do so increased by 12 percentage points between 2007 and 2010 according to internal HC analyses.

In addition to the evaluation of the Hazardcheck campaign, HC has also evaluated its national retail promotional events and collected purchase data from its retail partners. With respect to the most recent campaign in early 2013, 100,000 Hazardcheck guides and 3,000 Hazardcheck Checklists were distributed during seven weekend in-store retail events. The majority of visitors to the events who completed a feedback card (89 per cent) found the information they received to be very or somewhat useful. A follow-up survey was conducted of over 300 visitors. ^{Footnote #edN61} Based on the follow-up survey, almost all visitors found the Hazardcheck Guide and Hazardcheck Checklist very (50 per cent) or somewhat useful (47 per cent). Almost 90 per cent also found their conversation with the event representative useful. The post-event survey examined actions visitors took following their visit: 55 per cent had a conversation about what they had learned with family or friends and just over one in five purchased an item based on what they learned (e.g., CO detector, radon home testing kit). One in ten visited the HC website on Hazardcheck to obtain more information. Purchase data from two large participating retailers indicated that in locations where the events were held, sales of air quality devices increased compared to the same period the previous year by between 29 and 1,465 per cent for CO monitors and over 350 per cent for radon testing kits.

Post-campaign POR with First Nations and Inuit showed that between 2010 and 2013, there was no significant change in the proportion of First Nations (72 per cent, from 76 per cent in 2010) and Inuit (70 per cent, from 63 per cent in 2010) who reported taking some actions to protect against environmental hazards; however, more than one-third of First Nations (36 per cent) and one-quarter of Inuit (26 per cent) indicated they did something to protect against environmental risks after having seen the EHG campaign materials. The strength of confidence in their personal ability to take steps to protect their health from environmental issues also increased from 2010, with half of First Nations on-reserve (49 per cent) and Inuit (51 per cent) saying they are confident they can definitely take steps to protect their health (compared to 38 per cent and 31 per cent in 2010).

4.2.2 Increased awareness of data users

Finding 15. There is awareness among data users, particularly federal officials, of the monitoring and surveillance activities under the Action Plan -- the CHMS, FNBI, and (to a lesser extent) surveillance of development disorders/congenital anomalies.

Finding 16. There is evidence of awareness of the CHMS data among policy makers and researchers, but there is insufficient information to assess the extent of increased awareness. Researchers external to HC, PHAC, and Statistics Canada experience some challenges in accessing CHMS data.

The majority of data users interviewed for the roll-up evaluation, particularly federal officials, indicated that they are familiar with the monitoring and surveillance activities under the Action Plan -- the CHMS, FNBI, and (to a lesser extent) surveillance of development disorders/congenital anomalies. Awareness specific to the components is described below.

Canadian Health Measures Survey

The CHMS evaluation presented evidence of awareness among the general public, researchers and data users as a result of publications linked to Cycle 1. The evaluation further found that it is too soon to obtain valid measures of increased awareness since release of Cycle 2 is not yet completed.

Data for media uptake, CHMS website hits and requests for data, inquiries and custom tabulations are presented in the three tables below. We note that these data serve as benchmark measures of uptake and interest in CHMS data among researchers and the Canadian public against which subsequent cycles of CHMS can be assessed.

According to interviews conducted for the evaluation of the CHMS, awareness of the CHMS dataset varies. The evaluation found that there may be challenges in increasing awareness because of the lack of active promotion of the data to scientific communities. However, it was also noted by CHMS evaluation interviewees that the data are still relatively new and that awareness should increase as additional cycles are released.

The CHMS evaluation^{Footnote 21} found that the CHMS data are not as accessible as they could be, particularly for external researchers. Some of the challenges identified in the evaluation included the lack of availability of microdata files on the Internet and the fact that external researchers must go to Statistics Canada Research Data Centres (RDCs) to access the CHMS data. The CHMS evaluation also found that researchers experienced challenges with respect to the RDCs, including: a lack of awareness of how to access the data; timelines to get proposals approved; issues with locations and times of use; restrictions on material to be brought in and out; and the lack of knowledge of RDC staff about CHMS data files.

The CHMS evaluation found no challenges with respect to HC and PHAC researchers accessing CHMS data. Findings from the roll-up evaluation support this finding with internal data users identifying no challenges with respect to data access. HC and PHAC have a CHMS Data Share Agreement with Statistics Canada that provides HC/PHAC researchers with access to the data file for analytical purposes.

First Nations Biomonitoring Initiative

To date, the data collected via the FNBI has not been broadly circulated outside of First Nations communities that participated in the study. Data collection under the FNBI was conducted in a manner that took into account, where possible, First Nations protocols on data. AFN, a leading partner in the implementation of the FNBI, is the custodian of FNBI data. While access to the data is an aspect of the contribution agreement framework between HC and AFN, the details of how sharing of the data with other federal or external researchers will occur have not been finalized. The FNBI data are thus not yet fully accessible.

As mentioned previously, a national report was recently published (June 2013). It is anticipated that the data will be made available for broader release at the end of 2013-2014. As a result no specific performance data on awareness or take-up of the FNBI are available.

Congenital Anomalies and Developmental Disorder Surveillance

As noted previously, while there is national level information available on congenital anomalies (developed prior to the Initiative and which continues to be used for decision-making), there has been limited surveillance data collected or released by PHAC to address the enhanced systems outlined by the Action Plan and thus no progress with respect to this outcome to date. However, as with the FNBI, some data users are aware of surveillance activities and thus the future availability of these data.

4.3 Intended Longer-term Outcome

The intended long-term outcome of the Action Plan is for decision makers to use the information on the association between contaminants and illness to guide decision-making in public health practice, policy, regulation, programs and services. Ultimately, the Action Plan is expected to contribute to reduced health risks for Canadians (particularly vulnerable populations) from harmful environmental contaminants.

According to interviews with data users, the Action Plan monitoring data that are available are used and are important for a number of areas. As mentioned previously, there are linkages between the monitoring and surveillance components of the Action Plan and the CMP and CAA. Also within HC, the CHMS data are used for purposes such as regulatory, policy work and toxicology research.

4.3.1 Use of Data for Research and Decision-Making

Finding 17. Decision makers are using CHMS survey data to guide decision-making with respect to environmental contaminants, as well as other public health areas.

Findings from the CHMS evaluation demonstrated that the CHMS has been used for a variety of research, policy, and regulatory purposes. With respect to environmental exposures, the CHMS biomonitoring data have been published and have also informed recent updates to the science on exposure to lead and risk management of lead. There was further evidence that the information from the CHMS is being used for policy and decision making in the areas of physical activity, nutrition markers and oral health. The evaluation of the CHMS found evidence that CHMS data are being used within Statistics Canada in order to validate other surveys and to improve statistical modelling. Examples of how data from the CHMS survey have been used include:

• Human Health State of the Science Report on Lead, HC, 2013;
• Risk Management Strategy for Lead, HC, 2013
• Report on Human Biomonitoring of Environmental Chemicals in Canada: Results of the Canadian Health Measures Survey Cycle 1 (2007-2009), HC, 2010 (second report on Survey Cycle 2 is scheduled for April 2013)
• Canadian Mercury Science Assessment - Health Chapter, Environment Canada, (forthcoming, 2013);
• Canadian Environmental Sustainability Indicators, Environment Canada, 2012; and
• Report on the Findings of the Oral Health Component of the Canadian Health Measures Survey, HC, 2010.

External researchers surveyed as part of the CHMS evaluation confirmed that CHMS data are being used for a broad range of purposes, including: identifying priorities; making strategic and financial decisions; planning; updating guidelines and regulations; making international comparisons; and informing global negotiations. Key informants interviewed for the CHMS evaluation noted that CHMS data are viewed as being very credible among researchers and policy makers, and interviewees expect CHMS data to have a growing impact as additional cycles of data collection are added. These findings are supported by interview results from the roll-up evaluation. The roll-up evaluation asked users of monitoring and surveillance data to describe how they use the (CHMS) data. Examples of how CMHS data are being used by key informants include: for environmental assessments; to identify trends in exposure to environmental contaminants; for comparing exposure data across countries over time; to ensure scientifically valid information is available on effects of environmental contaminants/counter to less credible sources; and for validation of other data/survey results.

With respect to the FNBI, it is too soon to assess whether the data will be used for decision making since data have only recently been distributed. However, there is some evidence that the communities involved in data collection (i.e. those who have been included in the FNBI sampling) are making use of the data. In one community, for example, FNBI data are being used to develop health service programs.

As noted previously, the PHAC surveillance component of the Action Plan has not progressed sufficiently to measure or assess advancement towards the long term outcome.

4.3.2 Reducing Risks to Canadians

Finding 18. Action Plan components are beginning to contribute to reducing risks to Canadians from harmful environmental contaminants.

The Hazardcheck, and theFirst Nations and Inuit EHGs and companion documents are intended to make Canadians aware of the risks that harmful environmental exposures may pose to their health, along with the easy and inexpensive actions they can take to reduce these risks and improve their health. According to key informant opinion, the availability and use of the EHGs by Canadians can reasonably be expected to contribute to reducing risks to Canadians from harmful environmental contaminants.

Almost all of the key informants interviewed for the roll-up evaluation who were asked the question, indicated that the Action Plan components should lead to reduced risks to Canadians (particularly vulnerable populations) from harmful environmental contaminants. An early example of how Action Plan activities have resulted in reduced risks occurred in one of the communities selected for the FNBI pilot survey. The FNBI community results revealed elevated lead levels among some study participants, which led to community action involving HC, the AFN and local leaders to determine the source of the exposure. Community members were advised about the dangers of consuming foods hunted with lead pellets and steps in general to minimize their exposure to lead.

4.4 Enabling/Detracting Factors

Most EHGs program manager and staff respondents indicated that the Hazardcheck and First Nations and Inuit EHGs were delivered as designed. An exception was that while the First Nations and Inuit EHGs were to be "tailored" for First Nations and Inuit, in reality, these guides were developed concurrently with Hazardcheck and with unique content based on the needs and demographics of these populations. The strength of partnerships (e.g., with subject matter experts, First Nations and Inuit organizations) was most often mentioned as a positive external factor contributing to the success of the Action Plan public awareness component. For their part, partners were generally satisfied with their involvement in the Hazardcheck. For First Nations and Inuit EHGs, there were challenges cited by a few respondents in terms of engaging First Nations and Inuit organizations, agreeing on the key messages that were consistent with the HC evidence standards, and internal issues (such as lengthy HC approval processes, budgetary reductions and lack of coordination). This led to delayed development and distribution of First Nation and Inuit EHGs.

The evaluation data collection exercise for the PHAC surveillance systems provided a number of reasons for delays in implementing the surveillance systems, particularly the development disorders surveillance system (currently focused on ASD in youth and children). These delays stem from the unanticipated challenges associated with the complex nature of ASD and the need to coordinate systems with health and non-health sectors, including: the need to pool multiple data sources, given the absence of a single repository of proposed data elements; the lack of uniformity across jurisdictions for the identification of ASD; and, a varied state of P/T readiness for data capture and capacity.

A challenge identified by the CHMS evaluation is the lack of systematic tracking of publications that use CHMS data. As a result, there is little coordination and communication among researchers who are using the data. Better coordination and communication would allow researchers to know whether someone has already studied and is intending to publish results on their potential area of focus before starting their research.

Other challenges identified in the CHMS evaluation included: budgetary restrictions; limited knowledge transfer about the CHMS within RDCs; and "perception danger" occurring because if a trend cannot be observed at the present time, it may be perceived that measures are not worth the money - this is a problem because some trends take many years to become evident.

4.5 Unintended Impacts

Finding 19. There were few unintended impacts as a result of the delivery of the Action Plan. Positive unintended impacts included identification of medical issues at the level of individual participants, and collection of nationally representative data on chemicals of unanticipated importance. For the components of the Action Plan involving First Nations and Inuit, the partnerships that were established built trust and capacity, though expectations in some areas could not be met.

There were few significant unintended impacts identified. According to a few key informants, the components of the Action Plan that were targeted to First Nations and Inuit - the First Nations and Inuit EHGs and the FNBI - had some notable unintended impacts. For example, while partnerships with organizations such as the AFN are typical of the way the FNIHB structures its activities, and were perceived to be essential to complete a study such as the FNBI, a positive unintended impact was the further building of relationships and trust between HC and First Nations organizations and communities that participated as partners in the FNBI. There was also evidence that the FNBI increased capacity within First Nations to conduct biomonitoring studies, and raised interest in biomonitoring, both in Canada and in the United States. In addition, medical problems were identified in a few participants after direct measurements were taken as part of the FNBI, and immediate action was taken (e.g., hospitalisation for high blood pressure). Finally, the use of the First Nations and Inuit EHGs to support Aboriginal language use and instruction in communities was noted as an unintended impact by a small number of key informants.

The evaluation data collection exercise for the PHAC surveillance systems noted that bringing together members of Advisory Committees facilitated building of relationships for programs and activities in the areas of congenital anomalies and developmental disorders in the future.

According to the CHMS evaluation, in modeling the survey after the US NHANES--a survey that has existed for the last 50 years--there was an opportunity to take into consideration their experience and lessons learned, which, in turn, decreased the likelihood of unintended negative impacts. According to the internal interviewees, the complexity of the survey and the required extensive and thorough planning process, to ensure the controlled environment of operations and quality of data, further limited the likelihood of variations and unintended negative effects occurring and being observed.

Examples of unintended positive impacts of the CHMS included: 1) Some individual medical problems were identified and respondents were notified as soon as possible, which allowed them to seek advice from their doctor in a timely manner; and 2) Some environmental contaminant data that were collected as supplementary (i.e., selenium) in the CHMS became needed earlier than expected for risk assessment purposes.

4.6 Efficiency and Economy

The demonstration of efficiency and economy, according to the Treasury Board Policy on Evaluation (2009), is based on the assumption that departments have standardized performance measurement systems and that financial systems link information about program costs to specific inputs, activities, outputs and expected results. In general, financial information for the Action Plan was not compiled in a way that permitted this analysis (e.g., FNIHB use of a common financial code for separate Action Plan components, recording of CHMS funding by fiscal year rather than by cycle). Considering this issue, the evaluation provides observations on efficiency and economy based on findings from available relevant financial data and key informant interviews.

Planned and Actual Expenditures

Finding 20. Analyses of available financial data show significant variances between planned and actual expenditures related to surveillance activities and the EHGs in the early years of the programs. As the programs matured, these variances have generally decreased. CHMS and FNBI expenditures show relatively minor variances.

Table 10 summarizes available annual planned and actual spending on the Action Plan components. Obtaining complete and consistent financial data proved to be a challenge for several Action Plan components. The CHMS and FNBI planned and actual expenditures show relatively minor variances. The other Action Plan components show more significant variances, particularly in the early years of the programs. Where funds were unspent on Action Plan activities, information was not always available to determine where within departments they had been transferred or to confirm transfer approvals. According to program representatives, for the EHGs, in the first years there were delays in receiving funding, in staffing and in content development, as well as with obtaining guide and campaign approvals. In addition, federal elections precluded advertising activities. The program indicated that some variances also resulted from some program activities being conducted internally, as opposed to making use of contracted services. Program representatives also indicated that, in the case of Hazardcheck, Action Plan unspent funding in the early years was either used for related work by other programs, such as the Vulnerable Populations Program, or was lapsed for those fiscal years. As noted previously, due to unanticipated challenges, PHAC experienced delays in developing or enhancing the surveillance systems as initially planned. This led to variances in their planned and actual expenditures. As the Action Plan matured, variances generally decreased.

Cost-Efficiency Measures

Finding 21. The cost-efficiency analysis was hampered by limitations in the data on expenditures by component and outputs, as well as a lack of baseline data or international comparisons to provide contextual information.

A number of potential cost-efficiency indicators were proposed in the Action Plan evaluation framework to measure performance of the components, for example, cost per guide distributed for the EHGs and cost per person monitored for the CHMS and FNBI.

With respect to the calculation of the cost per guide distributed, there are two data limitations. First, on the cost side, the expenditures by the FNIHB for the EHGs for First Nations and Inuit and for the FNBI cannot be separated for the early years of the Action Plan as a common financial code was assigned for Action Plan expenditures for these two components. Similarly, expenditures by PACCB encompass both marketing for Hazardcheck and the First Nations and Inuit EHGs. Second, on the output side, the distribution statistics include the print guides only - web-site views or downloads of the guides online are not captured, thus, underestimating the actual extent of distribution. On the other hand, the First Nations and Inuit guides were distributed through EHOs and community health centres in which inventory may not have been completely depleted thus overestimating the actual extent of distribution.

Concerning cost per person monitored, again, the expenditure data for the FNBI are consolidated with the First Nations and Inuit EHG (in the first two years of the program). Calculating cost per person monitored for the CHMS is also difficult, as financial data are compiled by fiscal year rather than by cycle (number of persons monitored). Comparisons with the FNBI could be performed if the data were available; however, even then the coverage and administration of the surveys were somewhat different, which limit comparability. Moreover, there are no baseline data on resource utilization for surveys such as the CHMS and FNBI as they are unique surveys in Canada. Neither the CHMS evaluation nor the roll-up evaluation performed any international comparisons with similar surveys that could provide a context to assess monitoring and surveillance costs.

Proposed efficiency indicators for the PHAC surveillance system included costs to establish agreements with 13 provinces and territories (congenital anomalies) and cost per sentinel site or population-based surveillance system established (developmental disorders). Given the progress in the enhancement/development of the PHAC surveillance systems to date, these calculations could not be made.

Governance/Management of the Action Plan

Finding 22. No overarching governance or management mechanism for the Action Plan was planned or implemented, and this did not appear to negatively impact the delivery and performance of the four components.

As noted previously, the four component programs funded under the Action Plan were a collection of distinct initiatives. While there is a conceptual linkage among the initiatives in that they are addressing various aspects of environmental health, operationally, they are independent. There was no overarching governance or management mechanism for the Action Plan. Rather, the components were each managed by the responsible department, or through an existing interdepartmental mechanism or other structure.

Most internal key informants were of the view that additional governance at the Action Plan level was unnecessary and no significant negative effects were mentioned as a result of the independent implementation of the program components. A few key informants noted that the lack of interdependence was due, in part, to the timing of the implementation for the components and their maturity. For example, at the time the Action Plan was initiated, findings from the monitoring and surveillance activities were not available to inform the content of the public awareness materials. Thus, funding for the Hazardcheck and First Nations and Inuit EHGs specified that the content of the guides be based on existing science.

Efficiencies Gained from Delivery under an Action Plan

Finding 23. The coordination of methodologies and approaches between the FNBI and the CMHS created efficiencies for the implementation of the FNBI. The Action Plan was important to stabilize funding for the monitoring and surveillance systems, reducing the costs associated with diverting efforts to seek sources of funding for these types of longer-term commitments.

There were two areas where efficiencies were potentially gained by delivery of the components under an Action Plan. First, both the CHMS evaluation and the roll-up evaluation found that efficiencies were gained because the FNIHB and AFN were able to work in collaboration with Statistics Canada and benefit from the existing CHMS infrastructure and survey experience. According to the evaluation of the CHMS and interviews conducted with FNBI program management for the roll-up evaluation, the replication of the selection of chemicals for monitoring, and adoption of CHMS survey and sample collection protocols and procedures, laboratory contractual arrangements, and analytical and reporting structure created efficiencies for the FNBI at the operational level. The shared approach also facilitated comparability of results between the CHMS and FNBI. Although this collaboration could have occurred even in the absence of an Action Plan, the Action Plan did require that the FNBI be comparable to the CHMS.

A second efficiency gained from delivery of the funded components under an Action Plan identified by a few roll-up evaluation key informants relates to the level and stability of funding of the Action Plan components over the five year duration of the initiative. The five year and ongoing funding from the Action Plan stabilized funding for building the monitoring and surveillance systems, and reduced the transaction costs associated with diverting efforts to seek sources of funding for this type of long-term commitment.

Other Factors Supporting/Hindering Efficiency

Finding 24. A variety of efforts at the operational level have contributed to efficient delivery of public awareness, and monitoring and surveillance activities under the Action Plan. Some suggestions were made by key informants to continue to improve management structures and streamline protocols.

Considering the delivery of Hazardcheck and the First Nations and Inuit EHGs, key informants most often mentioned that delivery efficiencies were achieved through the extensive use of online tactics for both marketing campaigns. Internal analyses of the Hazardcheck social marketing campaign also indicated that online outreach tactics, including social media efforts, were found to be effective in driving traffic to the website, and the website and other web-based resources meet the expectations of Canadians to receive this type of information online.

At the regional level, the use of existing distribution mechanisms and partnerships for outreach (e.g., partnerships with other regions/other branches within HC, other departments, NGOs, professional organizations) was noted to contribute to efficient distribution of Hazardcheck. The creation of multiple products (e.g., the more streamlined Hazardcheck "Checklist" product, the EHGs for First Nations and Inuit youth), cross-promotion of environmental health topics within the guides and leveraging contributions from retailers were also mentioned by a small number of key informants as supporting efficiency.

According to a few key informants, factors detracting from efficiency of the Action Plan's public awareness activities were: delays experienced during the content development stage of Hazardcheck due to turnover in personnel and capacity issues of external partners; weak communications between headquarters and regions that undermined the opportunity for regions to effectively capitalize on the national Hazardcheck campaign; time-consuming management of internal relationships and federal financial processes; and under-utilization or under-performance of some of the Hazardcheck products and marketing tools.

With respect to the monitoring and surveillance components under the Action Plan, a key efficiency noted in the CHMS and roll-up evaluation is the comprehensiveness of the datasets, particularly the CHMS. The CHMS data are shared across a broad number of federal partners, as well as external researchers/academia, to achieve program and research objectives in various health-related areas. This includes research interests such as nutrition, oral health and physical activity (in addition to research on the connections between environmental contaminants and health). According to key informants, a comprehensive survey such as the CHMS minimizes or eliminates the need to establish expensive separate and independent survey infrastructures and operations to address each specific research need since the CHMS data are appropriate for multiple research and data needs.

CHMS evaluation interview findings provided further evidence of the program's effort to apply good stewardship and to use resources in an economical way. For example, selecting economical locations for trailers (using federal or public locations for a nominal fee over private, for-profit locations to minimize rental costs), and contracting out some services, such as maintenance and IT support, to local companies rather than sending maintenance personnel from head office. Efforts were also being made to try and minimize the overtime and travel costs of personnel. The CHMS evaluation notes that as the program matured, its operating structure was re-organized through mergers. This generated cost-savings through lower salary expenditures, while at the same time providing what was perceived to be a more efficient management environment. The current CHMS management continues to look for internal operational efficiencies through regular operational reviews.

The financial analysis conducted for the CHMS evaluation revealed that the mobile examination clinics used by the program to collect health measures and biological samples accounts for the program's most significant costs (including the trailer and medical and laboratory equipment). According to program managers and some experts, this cost was balanced against the need for a controlled environment to obtain highest quality data and the practice replicates the methods used internationally (for example, the US NHANES). Some external researchers, nevertheless, felt that the use of local facilities and hiring non-governmental staff are worth further consideration to enhance efficiency without compromising data quality.

In terms of the FNBI, some key informants noted that the remote location of many of the communities that were selected to participate in the FNBI had the effect of driving costs of the data collection upward. As well, investments in community consultations and presentation of results in the participating FNBI communities, while necessary and beneficial to engage communities and help them to interpret findings, are costly. However, in an attempt to improve efficiency, FNBI used regional staff as much as possible and piggy-backed data collection regionally to reduce travel and shipping costs. Finally, efficiency was hampered to some extent by federal guidelines that limited the ability to move funds from year to year, which made it difficult for the program to re-allocate funds from the consultation phase (years 1 and 2) that were under-utilized to the conduct of the survey (year 4).

The evaluation data collection exercise for the PHAC surveillance systems noted that the program has gained efficiencies by leveraging expertise at a low cost through the participation of experts on advisory committees that are run on a voluntary basis. A potential improvement to efficiency suggested by some key informants was to develop a broad, Agency-wide privacy impact assessment (PIA) (a requirement of data transfer agreements from provinces/ territories to PHAC). A standard or template PIA was proposed to make data transfer easier and less burdensome, although these key informants acknowledged that it is not known if this is a feasible solution.

4.6.1 Program Alternatives

Key informants did not suggest any alternative models for the delivery of Hazardcheck and First Nations and Inuit EHGs, nor was there evidence in the documentation or literature of alternative (more cost-effective) models. Key informants noted that efforts in the public awareness area have utilized an array of tools and approaches in the marketing and distribution of the guides, and have leveraged partnerships within the regions and with First Nations and Inuit organizations. Complementary activities have been conducted in parallel under the CMP to promote environmental health messages with intermediaries (e.g., schools, public health professionals).

With respect to monitoring and surveillance, again, few alternative approaches were suggested. A few key informants noted that there are other, less costly means of understanding the presence of environmental contaminants (e.g., monitoring industry chemical production, more focused community-level studies), however, these are viewed as complementary data sources, and not a replacement for nationally representative data. The CHMS evaluation reported that while all key informants believed that the CHMS should be funded by the federal government, some external researchers raised the question of whether the federal government should be conducting all activities versus contracting out some of the work. Alternative models for program delivery were suggested, such as completing some of the CHMS work by non-federal government contracts; however, this alternative was seen by others as having the potential to impact respondents' level of trust in the CHMS, as well as the CHMS's response rate and data quality.

4.6.2 Priorities Moving Forward

Public Awareness

Where there were suggestions from key informants to improve the delivery of environmental health messages, these most often had to do with increasing the use of social media as an inexpensive, yet effective tactic for fostering public awareness, both for the general public and First Nations and Inuit target groups. Other priorities moving forward, mentioned by one or two respondents included: greater partnering with other levels of government and non-governmental organizations within HC regions to increase the efficiency of the distribution of Hazardcheck; more interactive tools available to HC regional staff to engage the general public; and greater sharing of lessons learned from the experience with Hazardcheck and the First Nations and Inuit EHGs, while not negating the uniqueness of the each target group.

There were also some suggestions to modify and update the coverage of topics in the Hazardcheck and First Nations and Inuit EHGs and to refine the target groups based on emerging findings from the monitoring and surveillance activities on exposures and higher risk groups. Examples of new topics that were proposed included pesticides and wood smoke, as well as broader coverage of chemical substances. Both key informants and the international literature that was examined in the evaluation (Australia, Belgium) also identified climate change as a key theme for attention related to environmental health.

With respect to target groups, the review of documentation and literature indicated that attention to vulnerable populations such as children and Aboriginal/indigenous groups is a common feature of environmental health action plans in other jurisdictions. Vulnerable groups are typically identified based on socio-economic disparities, geographic location (e.g., living in areas with high levels of air pollution, nuclear waste), or physiological vulnerabilities. With respect to Canada's Action Plan, some key informants recommended focusing efforts on some specific target segments such as seniors (an outstanding Action Plan funding commitment), as well as segments such as first-time homebuyers/renters and owners of older homes who may be at higher risk. Development and dissemination of fact sheets for teachers and students is another outstanding commitment in the Action Plan that may merit consideration. Some of the work (e.g., companion documents/fact sheets for sub-populations, including for seniors, parents/caregivers of children, those planning a home renovation, tenants with concerns about mould, individuals with asthma or other respiratory disease) is underway and will be completed with ongoing funding under the Action Plan.

Monitoring and Surveillance

A priority for most key informants, including data users, is the continuation and expansion of monitoring and surveillance activities to support a robust understanding of Canadians' exposures to environmental contaminants, sources of exposures and the connections between environmental contaminants and human health. Again, attention to data collection (biomonitoring, integrated information systems for data sharing) is a priority area in environmental health action plans in other jurisdictions, including inquiry to understand the links between environmental contaminants and health effects, and identifying emerging issues. Based on the views of CHMS and FNBI program managers and data users, priorities moving forward for monitoring and surveillance in Canada include: through additional cycles, continued monitoring of the breadth of harmful chemicals, as well as chemicals that have not been examined before or that are new to the Canadian market; additional data collection to examine 'hot spots', higher risk groups, or longitudinal studies; inclusion of younger age groups as well as dietary and nutritional intake components for the FNBI.

Ongoing annual funding for the CHMS under the Action Plan ($14M) is being used for continued cycles of the CHMS (along with external cost-recovery funding) and will allow at least some of these priorities to be addressed. Ongoing funding under the Action Plan for the FNBI is more limited ($0.7M^{Footnote #edN62}, including $350K in grants and contributions funding available for an external partner organization such as the AFN), and much less than required for further data collection cycles or for addressing these priorities.

Coordination and Engagement

The international literature suggests that the successes of other countries in implementing environmental health action plans have been fostered through collaborative interdisciplinary and cross-sectoral efforts, and forging strong ties between science and policy. The European Union's successes, for instance, have included strong coordination and collaboration between health, environment and research sectors and consolidation of inter-sectoral initiatives, such as human biomonitoring and indoor air quality. This literature suggests that dedicated and sufficient resources for the task are also important, as well as engagement and communications with stakeholders, professionals and at the community level.

As discussed above, coordination among the Action Plan components was minimal, with the exception of the collaboration between the CHMS and FNBI components. The evaluation of the CHMS suggested that there is room for even more collaboration between these two initiatives.

The evaluation evidence indicates that a policy link was forged between the Action Plan monitoring and surveillance activities and the CMP (and to a lesser degree with the CAA). Engagement at the community level is realized through the EHGs, although the Action Plan public awareness activities have not benefited, to date, from monitoring and surveillance due to the timing of these research efforts.

5 Evaluation Conclusions and Recommendations

5.1 Conclusions

5.1.1 Relevance

There is a continuing need for federal efforts to protect Canadians' health from environmental contaminants, due to demonstrable risks or as yet unknown potential risks to human health from harmful environmental contaminants. There is also an economic and social burden attributed to negative health effects that are linked, at least in part, to environmental causes.

The Action Plan aligns with Government of Canada's environmental health priorities and there is a legitimate role for the federal government in its delivery. The Action Plan is consistent with federal roles and responsibilities in the area of health and contributes to fulfilling authorities laid out in legislation (such as the Canadian Environmental Protection Act, the Pest Control Products Act, the Food and Drugs Act, the Canada Consumer Product Safety Act) as well asthe federal government's responsibilities with respect to the health of First Nations. Additionally, it contributes to fulfilling Canada's obligations as signatory to a variety of international agreements pertaining to health and the environment. The federal role is also consistent with the expectations of stakeholders who have called for a greater commitment to monitoring and surveillance of environmental health, and reflects international practice, including other jurisdictions' national health measures surveys and implementation of national action plans on environmental health.

There is concurrence between the Action Plan components and the strategic outcomes and priorities of the three partner departments. The components of the Action Plan are also aligned with departmental mandates.

5.1.2 Performance

The determination of progress on the achievement of the Action Plan's intended outcomes is hampered to some extent by limitations in performance information. For example, due to government-wide restrictions on contracted public opinion research (POR), changes in Canadians' awareness of the health risks from environmental contaminants could not be fully assessed. Delays in the implementation of the PHAC surveillance systems also limited the assessment of outcomes in this area.

The public awareness component of the Action Plan has made progress in achieving its objectives. There is evidence that Canadians are aware of connections between environmental exposures and health (although targeted increases in awareness could not be fully assessed due to available data) and that some individuals have taken action to minimize environmental health risks. In terms of First Nations and Inuit, following the implementation of the Action Plan, individuals were more able to identify, without prompting, some form of environmental issues that affect health. Further, both First Nations and Inuit indicated an increased sense of confidence in taking steps to protect themselves from these environmental risks. Although increased public awareness and action were not objectives of the monitoring and surveillance activities conducted under the Action Plan, there is evidence that both CHMS and FNBI made contributions in these areas through interactions and sharing of biomonitoring results with individual participants (CHMS and FNBI) and with participating communities (FNBI).

Monitoring and surveillance activities have also made progress in achieving their stated objectives. Evaluation evidence indicates that there has been an increase in reliable and usable data with respect to understanding the connections between environmental contaminants and human health as a result of the CHMS and FNBI. Although progress has been made in establishing the developmental disorders surveillance system and enhancing the congenital anomalies surveillance systems, limited data was available at the time of the evaluation. Findings regarding awareness, accessibility and use of monitoring and surveillance data are mixed. For the CHMS, the data are well-known by federal researchers involved in a wide variety of research and policy areas and there is preliminary evidence that they are using the data to inform decision-making; some accessibility issues were identified for external researchers. For the FNBI, progress towards the longer term outcomes is currently limited as national level analysis and reporting just took place.

The demonstration of efficiency and economy, according to the Treasury Board Policy on Evaluation (2009), is based on the assumption that departments have standardized performance measurement systems and that financial systems link information about program costs to specific inputs, activities, outputs and expected results. In general, financial information for the Action Plan was not compiled in a way that permitted this analysis.

The perception among key informants was that the Action Plan represented good value for money and was delivered in an efficient manner, utilizing appropriate management, planning and partnerships to realize efficiencies. The Action Plan as an overarching funding mechanism did not appear to confer any significant efficiencies or inefficiencies in the delivery of the components, with the exception of the linkage between the FNBI and the CHMS, which served to increase the efficiency of the implementation of the FNBI (by capitalizing on the experience of the CHMS) and the comparability of the FNBI and CHMS data. Action Plan funding was also noted to have conferred some efficiencies to CHMS through stabilization of infrastructure and enabled Statistics Canada to continue with additional data collection cycles to address ongoing priorities in a variety of health areas. However, parallel FNBI data collection does not appear to be similarly funded.

Analyses of available financial data show significant variances between planned and actual expenditures related to surveillance activities and the EHGs in the early years of the programs. Where funds were unspent on Action Plan activities, information was not always available to determine where within departments they had been transferred or to confirm transfer approvals. As the programs matured, these variances have generally decreased. CHMS and FNBI expenditures show relatively minor variances.

There were no alternative models proposed to the current delivery of public awareness or monitoring and surveillance activities. As the components move forward with ongoing funding under the Action Plan, there may be a more compelling rationale for a closer relationship among the components as monitoring and surveillance activities mature and are able to offer greater insights for public awareness messaging, and information gleaned from public awareness activities can provide input into decisions on risk management. Lessons learned from public awareness activities for the general public and for First Nations and Inuit could also be better consolidated and shared to ensure better practices, while recognizing the unique needs of each target group. Greater use of regional partnerships and networks to capitalize on existing relationships was also noted as a potential area for greater coordination. Suggestions to improve efficiency included adjusting promotional tactics based on lessons learned from the marketing campaigns and the implementation of standardized templates to facilitate data sharing agreements.

5.2 Recommendations

This evaluation resulted in one recommendation for Health Canada, as the Action Plan lead, and one recommendation for the Public Health Agency of Canada surveillance component. The Statistics Canada component evaluation included recommendations for the CHMS (see Appendix C).

Financial Tracking

The assessment of whether program outputs were produced efficiently, or whether expected outcomes were produced economically was hampered by limitations in the documentation of expenditures. Obtaining complete and consistent financial data proved to be a challenge for several Action Plan components.

Recommendation 1:

All Action Plan partners should implement effective financial data tracking and monitoring processes to ensure financial accountability and facilitate future assessments of efficiency and economy, and to enable Health Canada, as the lead for the Action Plan, to develop annual financial summaries.^{Footnote 23}

Public Health Surveillance

The Public Health Agency has the responsibility for undertaking surveillance of congenital anomalies (in coordination with the Canadian Congenital Anomalies Surveillance Network) and developmental disorders. This includes data collection, analysis and reporting, as well as dissemination of findings. These activities are dependent on partnerships with provinces and territories. Considering the complex nature of disorders such as Autism Spectrum Disorders (ASD), there is a need to coordinate systems with health and non-health sectors.

Evaluation evidence indicates that the Public Health Agency's developmental disorders surveillance system (initially focused on ASD in children and youth) has not yet progressed to the state of actively collecting data from provinces and territories. While progress has been made in developing this system, data is not yet available. The enhancement of the congenital anomalies surveillance system is in a similar state.

Surveillance information is still needed to address gaps in the evidence base and to increase understanding of the occurrence and patterns about congenital anomalies and developmental disorders such as autism.

Recommendation 2:

The Public Health Agency should work with partners to ensure that timely and usable data is available for decision makers on congenital anomalies and autism spectrum disorders, linking the influence of environmental contaminants on these areas.

Canadian Health Measures Survey

The Statistics Canada component evaluation of the CHMS included three recommendations. These recommendations were addressed through a Management Response and Action Plan (see Appendix C).

Appendix A - Evaluation Matrix

Appendix b - CHMS Evaluation - Management Response and Action Plan

Recommendation 1

Relevance:

It is recommended that the management of CHMS enhance planning and external coordination with researchers and stakeholders--both internally within the federal government and externally with researchers from academia--with respect to content determination (content plan) and the planning of analytical research based on the content.

Statement of agreement or disagreement:

Management agrees with this recommendation.

Management response:

The evaluation reinforced management's strategic objective this fiscal year to develop a longer-term content plan for CHMS to ensure ongoing relevance. This work has already begun and will be completed within the next 12 months. In addition, the evaluation provided evidence that better planning for analytical work across all three federal departments--Statistics Canada, PHAC and Health Canada--would be a valuable new strategic objective. A Terms of Reference has been drafted for a new CHMS Analytical Working Group, which will be chaired by Statistics Canada and composed of representatives from PHAC and Health Canada.

Recommendation 2

Performance:

It is recommended that the management of CHMS increase awareness of the data and improve their accessibility by promoting CHMS to a wider audience of potential clients and users, and by providing support on how to use the data.

Statement of agreement or disagreement:

Management agrees with this recommendation.

Management response:

Increasing the use and awareness of CHMS is an important objective that has been clearly identified within the evaluation. Management agrees that more work can be done in this area and has developed three clear initiatives to better promote and support the use of CHMS. First, we have begun training additional staff with HSD's Client Services area to respond to external client service requests for information on CHMS and for custom tabulations using CHMS data. Second, management is developing a communications plan to increase awareness of both the CHMS data within the RDCs and the availability of the Biobank samples for researchers. Third, management is considering holding more workshops across the country to increase users' capacity to analyze CHMS data. This last strategy, however, requires resources to pay for travel and additional costs to conduct such workshops. Achieving this objective will depend upon our ability to identify sufficient resources to increase the number of workshops held over the next two years.

Recommendation 3

Performance measurement:

It is recommended that the management of CHMS improve the performance measurement system, as a tool, to systematically collect performance data and monitor the progress toward achieving its outcomes.

3a) Improving data
To accurately assess the achievement of the long-term outcome and the impact of CHMS, it is recommended that management of the CHMS ensure that a formal tracking, such as the Client Relationship Management System (CRMS), is in place for publications and studies based on CHMS data, to demonstrate their use. The information on publications using CHMS data must be shared on a recurrent basis with researchers and stakeholders to enhance further external coordination.

3b) Financial information
It is recommended that accurate financial information is made available by the Finance Branch to support CHMS performance data collection to demonstrate the level of efficiency over time for decision making and accountability reporting.

Statement of agreement or disagreement

3a) Management agrees with this recommendation.
3b) Management agrees with this recommendation.

Management response:

3a) Management already has an adequate tracking system to monitor publications and studies based on CHMS data; however, it has not been systematically shared with external researchers. As such, management is proposing to post this list on our external CHMS website and update it annually.

3b) Management is undertaking an operational review of the CHMS Operations Section to ensure that processes and practices are optimal. This review will be completed over the next six months and will lead to implementing performance indicators to measure the efficiency of the program.

Financial Branch Action Plan:

The Finance Branch will develop standard reports and financial indicators in support of all programs. The organization has detailed financial information already available that can be easily formatted to better address evaluation needs of programs and be made available on a timely basis. The determination of the format and elements to be measured will be developed in collaboration, and validated with program managers and the departmental evaluation group. The final recommendation will be presented and approved by the Administrative Practices Committee.

Appendix C - Detailed Description of Action Plan Component Programs

Program Profile - Environmental Health Guides

The objective of the environmental health guides (EHGs) is to make Canadians aware of the risks that harmful environmental exposures may pose to their health, along with the direct actions they can take to reduce these risks and improve their health. The primary vehicle for this component of the Action Plan is the development of a (main) EHG - Hazardcheck - as well as guides for First Nations and Inuit, and specific sub-populations (youth), and the implementation of marketing campaigns for these products.

Hazardcheck

Hazardcheck: Hazards in the Environment: What you can do!^{Footnote #edN63} was published on March 1, 2010, andis available in print version and online.^{Footnote #edN64} The guide provides information on common health risks from contaminants that may be present in the home environment, and simple and practical steps to take to reduce exposure to these hazards. Within Health Canada, the Healthy Environments and Consumer Safety Branch (HECSB) is responsible for developing the content of the guide and its companion documents, with the input of internal HC subject-matter experts for quality assurance.

The list below provides the table of contents for Hazardcheck.

Hazardcheck Table of Contents

What You Can Do: The Basics

1. Use household chemicals safely
2. Ventilate your home
3. Wash your hands often
4. Take off your shoes when you come inside

What You Can Do: Improving Indoor Air Quality

1. Avoid second-hand smoke
2. Keep carbon monoxide out of your home
3. Test your home for radon
4. Keep humidity levels in your home down to reduce the risk of mould

What You Can Do: Reducing Exposure to Lead

1. Lead in paint
2. Lead in plumbing

What You Can Do: Safe Use of Consumer Products

1. Check the latest news about possible health risks
2. Turn down the volume - noise from consumer products

In addition to the Hazardcheck publication, other related materials on environmental health have been developed and are available, including:

• interactive content on the web-site, including a Virtual House Tour and online Home Health Quiz ("Test your knowledge of common home health hazards");
• links to external resources, including YouTube videos presenting material on reducing exposure to radon, carbon monoxide, mould and lead.
• Related materials on environmental health that are planned by HECSB but not yet available include:
• companion documents/fact sheets for sub-populations, including for seniors, parents/caregivers of children, those planning a home renovation, tenants with concerns about mould, individuals with asthma or other respiratory disease.

First Nations and Inuit Environmental Health Guide

The First Nations and Inuit Health Branch (FNIHB), in consultation with First Nations and Inuit peoples, have created EHGs for First Nations and Inuit. The First Nations and Inuit Guides, Your Health at Home - What you can do!, were published on May 7, 2010 and April 11, 2011 respectively. The list below provides the table of contents for the First Nations and Inuit "at home" guides.

First Nations Your Health at Home Table of Contents

• Indoor Air
• Noise
• Drinking Water
• Wastewater / Sewage
• Food Safety
• Artistry / Hobbies
• Looking for Health Risks

Inuit Your Health at Home Guide Table of Contents

• Indoor Air
• Noise
• Drinking Water
• Food Safety
• Artistry, Traditional Crafts, Hobbies
• Looking for Health Risks
• Resources

In addition to these guides, supplementary materials have also been developed to address the needs and concerns of specific sub-populations and seasonal environmental health concerns. Companion documents include:

• The First Nations and Inuit Youth Guides - Live, Play and Learn - What you can do! - and Activity Booklets (for Kids and for Teens);
• First Nations Outdoor Guides: Spring/Summer - Your Health Outdoors. What you can do during spring and summer! and Fall/Winter - Your Health Outdoors - What you can do during fall and winter!.
• Inuit Outdoor Guides: Your Health Outdoors. What you can do during Open Water Season! and - Your Health Outdoors - What you can do during Ice Season!.

Development of the content of the environmental health guides, including key messages, language and wording and presentation was guided by a public opinion research program and consultations with First Nations and Inuit partners. Research included focus testing of creative materials, as well as baseline surveys to determine awareness and knowledge levels of environmental health.

Environmental Health Marketing Campaign

Within Health Canada, the Public Affairs Communications and Consultations Branch (PACCB) is responsible for producing, branding, marketing and disseminating the environmental health guides (Hazardcheck and EHGs for First Nations and Inuit). The publication of the environmental health guides has been accompanied by a variety of strategies to raise awareness of the link between health and the environment. Strategies to promote Hazardcheck included:

• On-line outreach (blogger content and outreach, Twitter and Facebook posts, QR codes) and Google Ad word buys to drive visits to the Hazardcheck web pages;
• Public engagement events (annually in 2011, 2012, and 2013 at retail locations), which included distribution of the Hazardcheck guides and Hazardcheck checklists, as well as informal information conversations between event representatives and visitors;
• Hazardcheck / Halte-O-Risques insertions in Parents Canada and C'est pour quand magazines; and
• Traditional media, including radio spots/public service announcements/print ads promoting Hazardcheck.

Similarly, the First Nations and Inuit environmental health guides are available on the Health Canada web-site and can be ordered in print version, and have been supported by a variety of strategies, including:

• Traditional media, including print PSAs and TV and Radio PSAs developed in collaboration with First Nations and Inuit youth as well as TV PSAs developed by the Aboriginal Peoples Television Network (APTN);
• On-line outreach through Google Adword buys and Facebook advertising;
• Social media outreach through posts made to the Healthy First Nations and Inuit Facebook Page;
• Community events (i.e. community information sessions hosted by EHOs and school events);
• The guides have also been distributed to First Nations and Inuit communities directly (e.g., to households, schools and community health centres); and
• Other activities have been undertaken to promote the guides and the links between environmental risks and health (e.g., Youth art contest with APTN).

Distribution of all environmental health guides is tracked by PACCB. In addition, some evaluation activities have been undertaken to assess the effectiveness of various aspects of the marketing campaign.

Budget

Within the $84.6M budget allocated to the Action Plan, $13.1M from 2008-2009 to 2012-2013 has been allocated for Health Canada for development and marketing of the environmental health guide.

Governance

Development of the content of the environmental health guide is distributed within Health Canada, with HECSB taking the lead for content development for the main guide and its companion documents, and FNIHB responsible for creating the content of the versions of the guides for First Nations and Inuit peoples. Within Health Canada, PACCB is responsible for the marketing campaign for both the Hazardcheck and guides for First Nations and Inuit. They are also responsible for monitoring up-take after the marketing campaign and beyond. RAPB is responsible for disseminating "Hazardcheck" through outreach, events and engagement of regional networks.

Program Profile - First Nations Biomonitoring Initiative (FNBI)

Background

Canadian Health Measures Survey

The Canadian Health Measures Survey (CHMS) is an ongoing project that was launched in 2007 by Health Canada in collaboration with Statistics Canada. The CHMS collects information, including a biomonitoring component, intended to create national baseline data on the extent of health issues such as obesity, hypertension, cardiovascular disease, exposure to infectious diseases, and exposure to environmental contaminants. The biomonitoring portion of the CHMS will support nationally representative data on human levels of environmental chemicals. Blood and urine specimens are collected and analyzed for a number of contaminants, including metals, PCBs, flame retardants, pesticides, insecticides, herbicides, and bisphenol A. The CHMS was not designed to provide data on First Nations on-reserve or Inuit communities. The purpose of the First Nations Biomonitoring Initiative (FNBI) is to collect the first nationally representative data of the baseline levels for environmental chemicals in First Nation adults living on reserve.^{Footnote 24}

Need for Biomonitoring in First Nations Communities
First Nations communities can be vulnerable to chemical exposure for a number of reasons^{Footnote #edN65}. For example, eating fish, marine mammals and wild game has cultural, spiritual, and nutritional significance, but these food sources also tend to have much higher concentrations of mercury and other persistent organic pollutants (POPs) (such as polychlorinated biphenyls (PCBs), pesticides, and polybrominated diphenyl ethers (PBDEs)). Toxins build up in these and other food sources through the process of bioaccumulation. In addition, many pollutants can travel long distances and accumulate in more remote Northern communities due to air transport, water currents, and climate change.

FNBI Implementation

The FNBI is a five year program (with limited ongoing funds) funded under the Action Plan. The Initiative is a health survey which seeks to establish baseline information through biomonitoring of human exposure to environmental chemicals for First Nations' communities across Canada (south of 60). The FNBI was developed to complement the CHMS, a national health measures survey in Canada that is representative of the Canadian population (excluding First Nations' peoples on reserve and Inuit). The FNBI is a partnership between Health Canada and the Assembly of First Nations (AFN), with data to be owned by the First Nations' communities.

This study was designed to help First Nations communities:

• Better understand their exposures to various toxic chemicals;
• Determine whether a person or a group has an unusually high level of a toxic chemical in their body;
• Identify groups that may experience higher levels of exposure; and
• Track trends in levels of exposure in First Nations populations over time.

The target population of the FNBI is First Nation individuals living on-reserve in Canada aged 20 and older at the time of collection. The results of this biomonitoring assessment will also help guide action by individuals, communities, and government(s) and inform future research on chemicals and health.

The FNBI collects information on the health and lifestyles of First Nations by means of direct physical measurements (such as height, weight, blood pressure), a household questionnaire, and the collection of blood and urine samples. The samples collected as part of the initiative are analyzed for environmental chemicals, including: metals, PCBs, brominated flame retardants, pesticides/insecticides/ herbicides, PFOS/PFOAs, BPA, phthalates, and cotinine. There are 97 chemicals measured in total as part of the FNBI analysis. Results of the analysis are reported back to the study participant.

Consultations

Prior to the implementation of a survey of this nature, a consultation process was conducted with the AFN to assess the interest of First Nations in participating in a national First Nations' specific biomonitoring study. A First Nations' specific biomonitoring project on a national level provides First Nations communities the opportunity to focus on specific environmental health related issues and provide baseline data for future biomonitoring work or research on the health impact of chemicals in the environment. A component of the consultations was to raise awareness on a wide range of environmental chemicals.

An all-Chiefs Resolution was passed on December 10, 2009, to further explore the concept of biomonitoring in First Nations communities^{Footnote #edN66}. A second all-Chiefs Resolution was passed on July 22, 2010^{Footnote #edN67}. This resolution supported the continued planning and implementation of the FNBI.

A series of outreach workshops were held across Canada and a series of six fact sheets were produced:

• General Background
• Goals of Biomonitoring
• Chemical Exposure
• Examples of chemicals most often studied in biomonitoring
• What you can do to reduce your exposure to toxins
• Step-by-step Biomonitoring Process

Pilot Project

In 2010-2011, a pilot project of the FNBI was implemented in two First Nations communities. The goal of the pilot project was to^{Footnote #edN68}:

• identify issues related to transportation and shipping in a remote versus non-remote community;
• establish procedures and processes needed to ensure the study ran smoothly;
• determine the amount of time required to conduct the different aspects of the study;
• test the appropriateness and effectiveness of different communications approaches;
• identify human resources requirements;
• estimate financial requirements of the study, especially in remote locations, and;
• estimate participation rates in First Nations communities for future planning.

A total of 257 First Nations participated in the pilot project, conducted between January 23 and February 23, 2011. The pilot study was conducted in Manitoba, and communities were chosen by the Assembly of Manitoba Chiefs. The two communities consisted of one isolated, difficult-to-access, rural community and one urban, non-remote community.

Full Survey

The full survey (collection of samples from participants) of the FNBI was implemented in 2011-2012. Over 500 First Nations participated as part of the full survey. A stratified random sampling of communities was designed with the following regional allocation below in Table 1 below^{Footnote #edN69}:

Analysis and Dissemination of Results

This stage includes notifying participants of their results, reporting back to communities on community-level results, and producing a National Report that includes a comparison of First Nations results with the general Canadian population as collected under the CHMS. These activities took place in 2012-2013.

Budget

Within the $84.6M budget allocated to the Action Plan, $5.6M from 2008-2009 to 2012-2013 has been allocated for Health Canada to conduct the FNBI^{Footnote #edN70}.

Governance

The FNBI Advisory Committee was established in May 2010 comprised of, but not limited to, the following groups and areas of expertise: Assembly of First Nations representative(s), doctor, Elder(s), Health Canada representative(s), regional First Nations representatives, traditional healer and youth^{Footnote #edN71}. The Advisory Committee provided guidance and advice to the FNBI Steering Committee on the design, conduct, data analysis, and communication of the FNBI. The Advisory Committee also ensured that First Nations protocols were respected.

Prior to the implementation of the FNBI in communities, a Community Research Agreement^{Footnote #edN72} was signed between the community and the AFN, in conjunction with a Band Council Resolution (signed by the Chief and Council) to grant consent to have the FNBI conducted on their reserve.

An ethical review of the FNBI was conducted by Health Canada's Research Ethics Board. Approval was obtained first for the pilot study, completed on July 25, 2011^{Footnote #edN73}. Approval for the full study was obtained on May 15, 2012^{Footnote #edN74}.

Program Profile - Canadian Health Measures Survey

Background

CHMS is Canada's first nationally representative direct health measures survey. Planning and development for CHMS started in 2003 as part of an extension of the Health Information Roadmap Initiative and in response to the need for a national, comprehensive source of accurate direct health data, which was expressed by policy makers, provincial health departments, researchers and health professionals from many fields. The survey was launched in 2007 to address long-standing limitations and data gaps within Canada's health information system.

To assist Canadians in reducing their health risks resulting from environmental contaminants and to further develop environmental health monitoring and surveillance, the federal government introduced the Action Plan to Protect Human Health from Environmental Contaminants (the 'Action Plan') in 2008. The Action Plan's long-term objective is to reduce health risks to Canadians, particularly vulnerable populations, from harmful environmental contaminants. To accomplish this objective, the Action Plan has two basic components: the first is the "environmental health guide" and the second is monitoring and surveillance. Statistics Canada's CHMS falls under the monitoring and surveillance component. This component consists of a series of surveys and surveillance activities, with the objective of providing Canadians with a better understanding of environmental exposure and potential related health risks.

Canadian Health Measures Survey objectives, activities and expected outcomes

CHMS collects key health information on Canadians through direct health measurements such as blood pressure, height, weight, and blood and urine samples. In addition, CHMS uses household interviews to gather information for other variables, including nutrition, smoking habits, alcohol use, medical history, current health status, sexual behaviour, lifestyle and physical activity, environmental and housing characteristics, as well as demographic and socioeconomic variables.

Objectives

CHMS has an important role in supporting the broader health agenda of the federal government.

The principal objective of the CHMS is to collect new and important data on Canadians' health status by

• providing a platform and infrastructure for obtaining data and information through physical and laboratory measures to meet the emerging needs of several branches within HC and PHAC, as well as other add-on studies
• collecting and disseminating direct health measures data, including those on environmental contaminants
• promoting research using direct health measures data by providing access to nationally representative data.

Activities

CHMS is designed and implemented in consecutive cycles, each of which consists of three activities: planning, collection and dissemination. Table 2 shows the CHMS life cycle.

Expected outcomes

The activities and outputs of CHMS are expected to lead to the following expected outcomes as specified in the Roll-up Evaluation Framework for the Action Plan to Protect Human Health from Environmental Contaminants.

Immediate outcome (1 to 2 years):

Reliable and usable data for decision makers, researchers and Canadians on the baseline health status of Canadians, and the level of, and exposure to environmental contaminants

Intermediate outcome (2 to 5 years):

Increased awareness among data/ information users of the collected data/ information

Long-term outcome (5 to 10 years):

Decision makers increasingly use the information on the associations between contaminants and illness to guide decision-making in public health practice, research, policy regulation and programs & services.

Leading to the ultimate outcome (10 to 20 years):

To reduce the health risks to Canadians, particularly vulnerable populations, from harmful environmental contaminants.

Budget

The initial funding for CHMS was allocated in the 2003 federal budget as part of an extension of the Health Information Roadmap Initiative. In 2007-2008, CHMS was funded $975,860 through an agreement between Statistics Canada and HC. This funding supported the completion of the CHMS planning activity for Cycle 1, prior to the launch of the Action Plan in 2008-2009.

Under the Action Plan, the total investment ($84.6 million over a five-year period) is divided between HC, PHAC and Statistics Canada. The source of funds attributed to Statistics Canada equals $54.5 million over a five-year period, which represents 64% of the total allocation of funds under the Action Plan. Based on this funding, CHMS was expanded to include additional measures on environmental exposure as well as direct health measures for children under the age of six.

The CHMS program received total core funding of $47.7 million for the period covered from 2008-2009 to 2012-2013. A total of $48.7 million has been allocated to CHMS from 2007-2008 to 2012-2013, inclusively.^{Footnote 25}

CHMS also received external funds from HC, PHAC and other entities to support special requests for additional laboratory tests and analysis that were not planned in the core funding. From 2007-2008 to 2012-2013, a total of $37.6 million was allocated as external cost-recoveries.

Therefore, the total CHMS resource allocation under the scope of this evaluation (i.e., core funding plus external cost-recoveries) represents $86.3 million over a five-year period and is detailed in Table 3.

Governance

Statistics Canada's Policy Committee, headed by the Chief Statistician and supported by the agency's assistant chief statisticians, is responsible for the administration of CHMS under the Statistics Act. In addition to the agency-level approvals and processes, the Canadian Population Health Statistics Program (CPHSP) Committee is responsible for ensuring that the Population Health Program at Statistics Canada responds to the health needs of all Canadians. CPHSP comprises assistant deputy ministers from HC, PHAC, and Statistics Canada.

HC and PHAC's joint Research Ethics Board (REB)^{Footnote #edN75} helps ensure that CHMS meets the highest ethical standards, and that the greatest protection is provided to participants who serve as research subjects. It is guided by the principles of the Tri-Council Policy Statement, Ethical Conduct for Research Involving Humans,^{Footnote #edN76} which sets the standard for research ethics boards in Canada. The REB reviews and approves the procedures and general conduct of the survey prior to the start of each CHMS cycle.

A privacy impact assessment is presented to the federal Privacy Commissioner for every cycle of the survey in compliance with the Privacy Act. Further, provincial privacy commissioners are advised of the survey collection activities within their jurisdiction and steps are taken to protect the privacy of survey participants according to applicable provincial law.

Since the inception of CHMS, four advisory committees have provided expertise and advice on various aspects of the survey: the Expert Advisory Committee, the Physicians Advisory Committee, Laboratory Advisory Committee, and Quality Assurance and Quality Control Advisory Committee.

In June 2012, the Biobank Committee was created. The Biobank Committee is chaired by a director general and consists of four members internal to the federal government and four external members (e.g., academics specializing in genetics or environmental contaminants). It is responsible for granting access to Biobank specimens, which are part of the CHMS data. CHMS management has recently taken the initiative to re-align the structure of the advisory committees with respect to the current, more mature stage of the survey.

Program Profile - PHAC Surveillance Systems

Background

PHAC's role within the Action Plan is to:

• Establish a new surveillance system for developmental disorders; and,
• Enhance the existing surveillance system of congenital anomalies.

The Surveillance Component of the Action Plan is comprised of two separate surveillance systems. PHAC's role is to provide surveillance of congenital anomalies and developmental disorders with a focus first on measuring Autism Spectrum Disorders (ASD) in children and youth, and eventually expanding to include adults and other developmental disorders. These two systems differ on a number of factors, although they share the same goal -- to inform on the impact of environmental contaminants on human health.

The Congenital Anomalies Surveillance System

The congenital anomalies surveillance system (CA) was set-up in 2002 under the Canadian Perinatal Surveillance System, then housed in Health Canada. The CA typically includes information on congenital anomalies identified during pregnancy, at birth or during an infant's first year of life.

In this initiative, the Public Health Agency of Canada will use the Canadian Congenital Anomalies Surveillance Network as a foundation to strengthen provincial and territorial surveillance systems with respect to congenital anomalies. Resources are being used to enhance surveillance systems where they now exist and to support new systems where there are currently none.

The Developmental Disorders Surveillance System

Canada currently has no national surveillance program to identify and track developmental disorders, such as sensory impairments, ASDs and attention deficit hyperactivity disorder. Consequently, there is a lack of reliable information about rates and trends for these conditions in general, and in association with environmental contaminants.

Under the Action Plan, PHAC started by establishing its capacity to address the surveillance system for developmental disorders, determining conditions to be monitored, creating a governance structure for tracking the capacity within provinces and territories (P/Ts) to implement surveillance systems and working with select P/Ts to set-up sentinel sites or population-based surveillance systems. PHAC will analyze data collected at the centres of surveillance expertise for rates of occurrence, trends and, in time, an association with environmental contaminants.

The selected sentinel sites or population-based surveillance systems will be funded so that data can be collected from a variety of sources e.g. hospitals, schools, and community physicians. The information will then be shared with PHAC in order to build a national database for analysis and reporting.

Budget

PHAC was allotted a budget of $11.4 million for the five year period from 2008-2009 to 2012-2013. This budget was fairly evenly split between the two surveillance systems. The developmental disorders surveillance system was allocated $5.5 million over a five year period - the budget growing from approximately $0.3 million in 2008-2009 to $1.7 million in 2012-2013. Similarly, the congenital anomalies surveillance system was allocated $5.9 million over a five year period. The budget here also grew from just over $0.3 million in 2008-2009 to $1.7 million in the 2012-2013.

Governance

Within PHAC, the chronic disease surveillance function falls within the Health Promotion and Chronic Disease Prevention Branch. Work under this branch is underpinned by three pillars of public health, namely to:

• Promote health
• Minimize risk for chronic disease
• Detect early and manage chronic health problems

The Integrated Strategy on Healthy Living and Chronic Disease (ISHLCDP), which falls under the Health Promotion and Chronic Disease Prevention Branch, is comprised of six functional components, one of which is surveillance, that work together to influence these three pillars of public health.

Apart from Health Canada and the Action Plan to protect Human Health against Environmental Contaminants, there is a variety of stakeholders, although the main data users would be public health leaders and decision-makers at the federal or P/T levels. Other beneficiaries would include anyone involved in evidence-informed policies, planning, programs and services at the local/regional levels, non-government organizations and academic researchers. As the two surveillance systems develop and grow, so will the range and number of beneficiaries.

Appendix D - Listing of Chemicals Monitored by the CHMS Cycle 1 and FNBI

List of Abbreviations and Acronyms

Item

Definition

ASD

Autism Spectrum Disorders

BPA

Bisphenol A

CA

Congenital Anomalies

CAA

Clean Air Agenda

CCDPC

Centre for Chronic Disease Prevention & Control

CDC

Centers for Disease Control and Prevention

CHMS

Canadian Health Measures Survey

CIHI

Canadian Institute for Health Information

CMP

Chemicals Management Plan

DD

Developmental Disorders

EHGs

Environmental health guides

EHMs

Environmental Health Managers

EHOs

Environmental health officers

FMS

Financial Management System

FNBI

First Nations Biomonitoring Initiative

FNIHB

First Nations and Inuit Health Branch

FTE

Full-time equivalent

HECSB

Healthy Environments and Consumer Safety Branch

IAQ

Indoor Air Quality

ISHLCDP

Integrated Strategy on Health Living and Chronic Disease Prevention

MIREC

Maternal-Infant Research on Environmental Chemicals Study

NCP

Northern Contaminants Program

NHANES

National Health and Nutrition Examination Survey

PAA

Program Activity Architecture

PACCB

Public Affairs Communications and Consultations Branch

PCBs

Polychlorinated biphenyls

PHAC

Public Health Agency of Canada

PIA

Privacy impact assessments

POPs

Persistent organic pollutants

POR

Public opinion research

P/T

Provincial / Territorial

RAPB

Regions and Programs Bureau

RDCs

Research Data Centres

REHMs

Regional Environmental Health Managers

RMAF

Results Management Accountability Framework

RPP

Reports on Plans and Priorities

SSA

Sub sub-activity

TB

Treasury Board

WHO

World Health Organization