Final Audit Report - Audit of the Management of Scientific Research

(PDF Version - 239 K)

March 2013

Table of Contents

Executive summary

The focus of the audit was on the management of scientific research at Health Canada. The Department has a responsibility to perform research and related scientific activities, and to assess scientific knowledge in order to apply a high standard of scientific evidence to departmental policy, regulatory and health protection activities and health programs.

The objective was to assess the effectiveness of the control framework for the management of scientific research in support of the Department's regulatory and policy mandate as it relates to governance, risk management, research selection, monitoring and reporting. Scientific research, in the context of the audit, is defined by management as the development of new scientific knowledge. The audit was conducted in accordance with the Treasury Board Policy on Internal Audit and the International Standards for the Professional Practices of Internal Auditing. Sufficient and appropriate procedures were performed and evidence gathered to support the audit conclusion.

At Health Canada, five branches are involved in the management of scientific research; however, two branches are primarily responsible for conducting scientific research to support the regulatory mandate: Health Products and Food Branch and Healthy Environments and Consumer Safety Branch (approximately 88 percent of the expenditures). These two branches conduct scientific research related to health products, food, environmental health, consumer product safety, and radiation protection. The Pest Management Regulatory Agency contributes to the design of scientific research and uses scientific research produced by other partner organizations to support its regulatory mandate. The Regions and Programs Bureau supports scientific research in its regional laboratories and the First Nations and Inuit Health Branch provides funding for scientific research. For last fiscal year there were approximately 221 departmental scientific research projects underway.

The management of scientific research is the responsibility of the branches. Currently, the processes within the branches vary concerning the governance, the establishment of research priorities, the selection of research projects, and the monitoring and reporting of performance information. Some practices are highly developed and well managed whereas others could be strengthened. It was found that while the individual research projects were sometimes well described, there is a need for better consolidation of information directly related to the scientific research either being planned or in progress. This would include a roll up of information including: area of interest, total funding by source, total resources required, alignment with risks and priorities and performance measures (including timelines).

There are three recommendations related to enhancing governance for scientific research (including an agreed upon branch definition for scientific research), strengthening the planning and reporting by allocating resources and tracking expenditures; and lastly, by developing performance measures to better assure the branches that the scientific research conducted and used will continue to advance the Department's regulatory and policy mandate to protect the health and safety of Canadians.

A - Introduction

1. Background

Health Canada's scientific mission is to perform research and related scientific activities, and to assess scientific knowledge in order to apply a high standard of scientific evidence to departmental policy, regulatory and health promotion activities, and health programs. Scientific research allows Health Canada to anticipate and respond to health risks posed by diseases, environmental hazards, food and water-borne diseases and other threats; to verify that the food, biologics, pest control products and consumer products are safe and effective; and to provide information to Canadians to help them make informed decisions about their health. Health Canada also relies on external research (academia, other government departments, etc.) to inform its regulatory mandate. Scientific research is fundamental to the Department; as such scientific research projects pursued must be prioritized, funded and managed to best position the Department to continue to protect the health and safety of Canadians.

Scientific research, in the context of the audit, is defined by management as the development of new scientific knowledge in a Health Canada-managed laboratory or an externally funded facility (including field research). It does not include routine testing, analysis, standard setting, surveillance, or the use of existing information.

The Healthy Environments and Consumer Safety Branch (HECSB) mandate is to help Canadians to maintain and improve their health by promoting healthy and safe living, working and recreational environments and by reducing the harm caused by tobacco, alcohol, controlled substances, environmental contaminants, and unsafe consumer and industrial products. The scientific research work in this Branch is divided between three directorates. Last fiscal year, the Branch reported 104 scientific research projects with expenditures of approximately $7.1 million.

The Health Products and Food Branch (HPFB) is the scientific and regulatory authority for health products and food. This Branch's mandate is to take an integrated approach to the management of the risks and benefits to health related products and food by minimizing health risk factors to Canadians, while maximizing the safety provided by the regulatory system for health products and food; promoting conditions that enable Canadians to make healthy choices; and providing information so that they can make informed decisions about their health. The scientific research work in this Branch is divided between two directorates. Last fiscal year the Branch reported 88 scientific research projects, with expenditures of approximately $13.7 million.

The Pest Management Regulatory Agency (PMRA) mandate is to prevent unacceptable risks to human health and the environment through the regulation of pest controls products. Under the Pest Control Products Act, the PMRA: regulates pest control products for use in Canada; develops policies and guidelines; promotes sustainable pest management; looks to improve the regulatory process to increase efficiencies; and carries out compliance and enforcement activities with respect to the law. For research and monitoring priorities not anticipated to be fulfilled by other jurisdictions or international partners, PMRA identifies priority needs to the other government departments. These departments consider these needs in their own operational planning, where possible. While PMRA's lab is primarily used for regulatory testing it may conduct occasional research into analytical methods development. (See page 20)

The First Nations and Inuit Health Branch (FNIHB), Environmental Health Research Division supports environmental public health, including drinking water safety, food safety and air quality, as well as research into the health implications of climate change and environmental contaminants in First Nations and Inuit communities. This scientific research is coordinated and funded primarily in one directorate, previously known as Primary Health Care and Public Health (recently changed to Interprofessional Advisory and Program Support Directorate). This Directorate funds approximately 8 community-based scientific research projects, totalling nearly $1.8 million aimed at assisting First Nations and Inuit in assessing the extent of potential health impacts resulting from exposure to environmental pollutants. For 2012-13, an additional 16 community-based scientific research projects worth $1.9 million will be funded.

The Regions and Programs Bureau (RAPB) contributes to improving and maintaining Canadians' health by effectively delivering programs and services across the country. This Bureau manages laboratories across the country ensuring collaboration across business lines and providing broad based analytical and scientific support to all regions, and to the Department as a whole, as well as to external clients, including law enforcement agencies. The laboratories include the Drug Analysis Services (DAS) Laboratory, the Food Program Laboratory, and the Inspectorate Laboratory Programme. RAPB reported that, for last fiscal year, the Bureau contributed to 12 Food Directorate scientific research projects and completed 5 other research activities which led to the development of new methodologies to support the Inspectorate and DAS laboratory activity. The total expenditure was approximately $1.0 million.

The Strategic Policy Branch (SPB) promotes the integration of science and evidence into departmental decision-making and assists the science branches in identifying areas which will require coordination for departmental and portfolio science policy.

While each of the above branches contributes to the Department's science mandate, the majority of the Department's scientific research is conducted by Healthy Environments and Consumer Safety Branch and the Health Products and Food Branch. Health Canada has approximately 100 research scientists who lead teams of chemists, biologists, engineers, and technicians in scientific research projects. In addition, the Department is able to direct priorities for external targeted research through arm's length governance structures responsible for health research.

Below is a departmental summary table of the scientific research conducted at Health Canada. The table notes the approximate dollar amounts for research that is conducted via Health Canada scientists versus that which is conducted by third parties through contracts.

Scientific research conducted at Health Canada
Responsible branch Directorates Research expenditures (in millions) % of total dollars
Internal External Total
HPFB
  • Food
  • Biologics, Genetics and Therapies
$13.7 $0.0 $13.7 58.1%
HECSB
  • Environmental and Radiation Health Sciences
  • Safe Environments
  • Controlled Substances and Tobacco
$5.1 $2.0 $7.1 30.1%
FNIHB
  • Primary Health Care and Public Health
$0.0 $1.8 $1.8 7.6%
RAPB
  • Regional laboratories (Drug Analysis Service (DAS), Food Program Laboratory, Inspectorate Laboratory Programme)
$1.0 $0.0 $1.0 4.2%
PMRA   $0.0 $0.0 $0.0 0.0%
Total $19.8 $3.8 $23.6 100.0%

*Branch research expenditures are for fiscal year 2011-12 and are unaudited.

2. Audit objective

The objective of the audit is to assess the effectiveness of the control framework for management of scientific research in support of the Department's regulatory and policy mandate as it relates to governance, risk management, research selection, monitoring, and reporting.

3. Audit scope

Scientific research, in the context of the audit, is defined by management as the development of new scientific knowledge in a Health Canada-managed laboratory or an externally funded facility (including field research). It does not include routine testing, analysis, standard setting, surveillance, or the use of existing information.

The audit focused on the management of the scientific research of two branches most actively engaged in scientific research: Healthy Environments and Consumer Safety Branch and the Health Products and Food Branch. At the time of the audit, these two branches accounted for approximately 88 percent of the Department's scientific research expenditures. The third key science branch is the Pest Management Regulatory Agency. Although the Agency has many science related activities, it does not conduct its own scientific research. PMRA makes research requests to other government departments and agencies, which carry out the research projects subject to their operational requirements. A separate section of the report was prepared to report on PMRA's management of its scientific research in this business model.

4. Audit approach

The audit was designed to carry out analysis, review and testing of key components of expected management controls for the scientific research activity. Audit work related to governance, planning, risk management, project selection and performance measurement was conducted via interviews, reviewing and analyzing documentation, policies, standards, guidelines, comparative analysis as well as a financial analysis of operational resources.

The audit criteria were derived from the Treasury Board of Canada Secretariat's Audit Criteria Related to the Management Accountability Framework. (See Appendix A) The audit was undertaken by the Portfolio Audit and Accountability Bureau in accordance with the Health Canada Risk-Based Audit Plan 2009-10 to 2011-12.

The findings and recommendations section of the audit report were divided into three chapters, each representing one of the three scientific branches.

5. Statement of assurance

In the professional judgment of the Chief Audit Executive, sufficient and appropriate procedures were performed and evidence gathered to support the accuracy of the audit conclusion. The audit findings and conclusion are based on a comparison of the conditions that existed as of the date of the audit, against established criteria that were agreed upon with management. Further, the evidence was gathered in accordance with the Internal Auditing Standards for the Government of Canada and the International Standards for the Professional Practice of Internal Auditing.

B - Findings, recommendations and management responses

Part I - Healthy Environments and Consumer Safety Branch

The scientific research work in this Branch is divided between three directorates. Last fiscal year the Branch had 104 scientific research projects underway, with expenditures of approximately $7.1 million.

The Environmental and Radiation Health Sciences Directorate had expenditures for 2011-12 of approximately $4.9 million. During that same fiscal year the Directorate had approximately 69 research projects underway between four bureaus: Chemicals Surveillance Bureau; Consumer and Clinical Radiation Protection Bureau; Environmental Health Science and Research Bureau; and the Radiation Protection Bureau.

Research Projects by Directorate
Percentage of Total Project Count

Controlled Substances and Tobacco Directorate: 8%, Environmental and Radiation Health Sciences Directorate: 66%, Safe Environments Directorate: 26%

The Safe Environments Directorate had expenditures for 2011-12 of approximately $1.7 million and had 27 research projects underway between three bureaus: New Substances Assessment and Control Bureau; Existing Substances Risk Assessment Bureau and the Water, Air and Climate Change Bureau.

The Controlled Substances and Tobacco Directorate had expenditures for 2011-12 of approximately $0.5 million for 8 scientific research projects.

1. Governance

1.1 Governance structures

Audit criterion: Governance structures should set priorities and integrate scientific research into branch decision-making processes.

Governance decisions should play a fundamental role in strategically directing scientific research. It would be expected that scientific research priorities would be tabled for decision at a senior committee and that results from the research would be integrated into the Branch decision-making process.

The Healthy Environments and Consumer Safety Branch has its Branch Executive Committee which is its senior most decision-making body, chaired by the Assistant Deputy Minister and is composed of senior branch management. On the scientific research front, this Committee is supported by an Executive Subcommittee on Science Policy chaired by the Director General Environmental and Radiation Health Sciences Directorate who also acts as the Chief Scientist for the Branch.

In addition, one of the bureaus, the Environmental Radiation Health Sciences Directorate has its Research Steering Committee which is comprised of the managers of the four divisions within the directorate and its Internal Review Board whose mandate is to review new project proposals or progress on existing projects. In another directorate - Controlled Substances and Tobacco Directorate has the Office of Research and Surveillance. This Office supports the Directorate by using a variety of mechanisms to identify research activities, ensures they are of high quality and generates valid and credible information. Scientific research related to the chemicals is governed via the Branch committees as well as the Chemicals Management Plan Monitoring and Surveillance Network.

The audit team reviewed relevant governance documentation and noted good evidence of agenda setting with each committee. While at the directorate level, where research projects are identified and initiated, the management teams are informed of scientific research project details, there was not a formal mechanism in place for advancing this information to the Branch Executive Committee for decision-making purposes. Depending on the circumstances, decisions are taken on a case by case basis or through such exercises as the operational planning process.

Currently the Branch is in the process of developing a new scientific research governance model which will use the good practices from each of the directorates. The new governance model will provide more formal and consistent decision-making across each of the directorates. This exercise also includes streamlining of the Chemicals Management Program governance model.

Recommendation 1

It is recommended that the Assistant Deputy Minister, Healthy Environments and Consumer Safety Branch implement an enhanced branch level governance for scientific research supported by the program governance.

Management response

Management agrees with the recommendation.

The Healthy Environments and Consumer Safety Branch had made significant strides in engaging Branch management through the establishment of the Branch Research Governance Committee, which reports to the Assistant Deputy Minister and the Branch Executive Committee, in October of 2012. The mandate of the Committee is to provide advice and broad procedural oversight on Branch science governance processes by:

  • reviewing governance practices for all research, monitoring and surveillance and research-related activities within the Branch;
  • developing a framework of recommended best practices for Branch research governance;
  • providing a forum for ongoing communication on governance practices in Healthy Environments and Consumer Safety Branch;
  • providing a forum for communication on priorities and results of research for information sharing and to identify areas of overlap and strategic collaboration;
  • supporting and responding to recommendations resulting from audits and programmatic peer review; and
  • establishing ad hoc working groups, as required.
1.2 Planning

Audit criterion: The branch should have strategic and operational plans for scientific research.

Setting out a strategic direction for scientific research should assist branches in reaching their operational research goals and in strengthening partnerships between those who conduct research and those who implement its findings. By integrating scientific research into the planning exercise, this may be useful in supporting: priority setting; allocation of resources for projects (funding and staff); and reporting.

A large amount of documentation related to science planning exists. There are Branch science plans, program specific science plans, research project abstracts, operational plans (corporate, branch, directorate and program), strategic human resources science plans. These plans could be strengthened by including more information on the specific research projects, risk analysis and associated funding.

The Branch documents its operational planning using the Integrated Planning and Performance Reporting System (IPPRS), which promotes uniformity in planning. Branches collect data in the IPPRS in a single coordinated process where they are to identify activities as well as supporting resources. The Branch has begun to incorporate scientific research into program planning which will support partnerships between those conducting the research and those using its results. Costs for existing staff are not included in this exercise. Salary costs are only required for contractors, students and post-doctorate researchers and are anticipated at the start of each project. Within one of the Branch's directorates, Environmental and Radiation Health Sciences Directorate, research templates are completed to capture both operating and salary expenditures associated with each project. The Branch would benefit from including the salary costs of all project staff, which will allow for better management of the full cost of research projects. In addition, an analysis of the template shows the inclusion of a variety of General Ledger cost categories. While the Branch guides managers to capture costs such as language training to their projects it does tend to inflate the actual cost for research projects. The Branch notes that it is a relatively simple exercise to extract the overhead costs from the actual costs for individual projects.

The Healthy Environments and Consumer Safety Branch has produced a thorough 2011-12 research compendium as a reference tool. The document provides research summaries being undertaken for last fiscal year. The Branch notes that this is a research related activity listing and was not intended to include a prioritization of the activities listed or to earmark the associated financial and human resources or research results. Identifying the anticipated results, associated financial and non-financial resources, as well as funding sources, would assist in justifying the scientific research investment.

Annually, Health Canada reports as part of the Federal Science Expenditures and Personnel Survey (CANSIM 358-0142 to 358-0151). The Survey collects data which are essential to assure the availability of pertinent statistical information to identify expenditures on research and experimental development; to monitor science and technology related activities in Canada; and to support the development of science and technology policy. The data collected also form part of the Gross Domestic Expenditures on Research and Development. However, through interviews, it was determined that the definition of scientific research is widely interpreted. This was confirmed following a detailed financial analysis, which showed a variance between the audited expenditures and those reported for the purposes of the Survey. The Branch should agree upon the interpretation of what constitutes scientific research in order to facilitate more accurate and reliable annual reporting.

The management of scientific research can be strengthened by establishing an agreed upon definition for scientific research, allocating resources, monitoring project progress and expenditures leading to more accurate and reliable reporting. This will help the Branch to secure a scientific research portfolio that demonstrates the most effective investment of research dollars as well as a portfolio that is responsive to emerging and important public health needs.

Recommendation 2

It is recommended that the Assistant Deputy Minister, Healthy Environments and Consumer Safety Branch build on existing practices to:

Management response

Management agrees with the recommendation.

The Assistant Deputy Minister, Healthy Environments and Consumer Safety Branch supports the establishment of Branch and program specific research plans. The Branch will continue to utilize the Integrated Planning Performance Reporting System (IPPRS) to track the allocation of funding each planning cycle.

Effort has been made to utilize IPPRS to track program level research initiatives (for example, Clean Air Regulatory Agenda (CARA), Genomics Research and Development Initiative (GRDI), Chemicals Management Plan (CMP), within the detailed Directorate plans. This has proven helpful in facilitating the Branch's ability to improve the accuracy of performance tracking and reporting.

The appropriate forum for the discussion of Healthy Environments and Consumer Safety Branch scientific research plans is the Branch Research Governance Committee (BRGC). A key component of the Committee's efforts is to establish a formal, Branch-wide definition of research. All plans will be held to the Committee's standards, mandate and authority. The BRGC also serves as the appropriate forum to facilitate knowledge translation, transfer and exchange efforts across the Branch.

2. Risk management

2.1 Integrating risk into scientific research

Audit criterion: Branch program risks should direct scientific research activities.

Scientific research should contribute to greater knowledge towards mitigating Branch program risks. Regular review of scientific research and its results should inform the risk process. As such, research objectives should be linked to reducing the identified health science risks.

In reviewing the documentation provided, it was noted that the Healthy Environments and Consumer Safety Branch developed a risk registry relating to the high level Branch risks. As well, in 2010, the Branch hosted a facilitated meeting to identify potential priorities for the Chemicals Management Plan/Canadian Environmental Protection Act/Research and Monitoring and Surveillance. From this, a list of short, medium and long-term priorities were identified. In addition, the Chemicals Management Plan team produced an integrated model, linking research to the other related activities such as risk assessment, compliance, promotion, enforcement, monitoring, surveillance and reporting. The intent is to improve integration and collaboration, and to deliver the science to support decision-making. The organizations involved recognized that there are continuing challenges in securing management priorities early, linking risks to these priorities and integrating findings into Branch operations.

The Branch has started to plan scientific research based on the data needs and to incorporate the results of research into subsequent risk assessment reports. There are also workshops conducted, and the assistance of international experts, who help to identify priorities for scientific research and, subsequently share the results of research and policy. Lastly, the Environmental Radiation and Health Sciences Directorate has recently identified key priorities, the associated activities, and funding sources in support of risk mitigation.

While there have been several risk-related initiatives, the Branch would benefit from clearly linking the scientific research as a mitigating strategy towards improving public health and safety. Secondly, the Branch would benefit from establishing that the research level of effort and resources are commensurate with the risks identified. (See Recommendation 2)

3. Project management in support of scientific research

3.1 Scientific research project selection

Audit criterion: Research projects should be aligned with the branch mandate and with established criteria for project selection.

Scientific research project selection and the associated funding should be the "official record" of what was decided by senior management. Annual Branch operational planning should be used to guide the research selection process and to record the financial amounts committed throughout the life cycle of the research project.

Directorates within the Healthy Environments and Consumer Safety Branch indicated that the prioritization of new research projects and the re-validation of research projects underway occurs during the annual operational planning process and is guided by Branch priorities. The Branch has started to integrate scientific research findings with project selection in a number of areas. These include new substances and micro-organisms, and the Chemicals Management Plan. The Branch also has a process in the Chemicals Management Plan to share scientific research results that impact other science mandates of the Department. In these cases, there is a confirmation of collaboration established at the start of the project. In addition there is evidence of external reviews of the project proposals which provide information for management decisions.

The Branch uses project and program peer review as standard tools in the assessment of scientific research. Joint ventures and Treasury Board submissions have more up-front clarity in project selection. Recently, the Environmental Radiation and Health Science Directorate developed a research template which has been further refined for use during the 2012-13 fiscal year. The ten page template includes sections dealing with project proposal information including timelines, objectives, rationale, work plan, costing and funding sources, milestones and deliverables. It includes a section on knowledge generation and transfer which identifies new information expected from the project (how it is different from what has been done before), how it will contribute to the Bureau's mandate and how there will be knowledge transfer from the project. The template requires the identification of operating and maintenance, capital costs and new salary costs by fiscal year. Progress reporting is done on a quarterly basis and there is a requirement for an end of project report. This template is not yet used consistently across the directorate or throughout the Branch but once fully implemented it will be a good internal control tool. In certain instances, an alternate template is required based on the project selection processes involved in a joint venture or for an external client. (see Recommendation 2)

4. Performance measurement

4.1 Performance measurement

Audit criterion: Scientific research should have performance measures which are monitored and reported.

Branch strategic direction for its scientific research should be supported by the research work performed and results obtained. A performance measurement framework for scientific research is an integral part of managing for results.

A recent horizontal evaluation was conducted on phase one of the Chemicals Management Plan. It noted concerns that performance information focused on research project activities and outputs, which informs management decision-making at the operational level. However, the evaluation also identified a need for additional information to inform strategic decision-making, planning, direction setting and performance reporting. This includes the need for information on the status of key activities and the identification of significant issues or delays and tracking of progress in key outputs and outcomes.

In response, the Directorate developed a new performance measurement strategy to "assist program managers and deputy heads to continuously monitor and assess the results of programs as well as the economy and efficiency of their management; make informed decisions and take appropriate, timely action with respect to programs; provide effective and relevant departmental reporting on programs; and ensure that credible and reliable performance data are being collected to effectively support evaluation."

The document clearly identifies the challenges associated with the development of performance measures for research projects. They include activities that span a long timeline before ultimate outcomes are realized, technical challenges in effectively measuring outcomes and technical and logistical challenges relating to the number of chemicals involved and the associated methods and costs. The document includes expected deliverables that are specific, measureable, and results based. There is a detailed evaluation strategy which lists outputs and outcomes, performance indicators, targets, data sources, frequency of data collection and responsibility for data collection. This is supplemented by an extensive number of specific performance measurement questions related to the program.

The Environmental Radiation Health Sciences Directorate has a process in place to identify the performance measures associated with its research projects. Within each project proposal there are milestones, deliverables and expected timelines. These are done for each quarter of each funding year. The template lists such examples as: data collection, data analysis/syntheses, publications, conference presentations, decisions taken, and knowledge transfer activities.

The extent of performance measures varies between directorates. Clearly, those research projects that are linked to Treasury Board funding, such as the Chemicals Management Plan, tend to have a better performance measurement framework.

While there was some evidence of performance measurement and it is acknowledged that scientific research is typically conducted over the long-term, there is still the need for appropriate qualitative and quantitative performance measures associated with scientific research in order to demonstrate appropriate management of scientific research expenditures.

Recommendation 3

It is recommended that the Assistant Deputy Minister, Healthy Environments and Consumer Safety Branch extend the performance measures to all its scientific research.

Management response

Management agrees with the recommendation.

The new Branch Research Governance Committee (BRGC) has a mandate related to the development of such a performance management framework. Previous efforts are being examined as case studies to help design and steer this initiative. It will be important to ensure that any performance measurement framework established at the Branch level does not duplicate or conflict with program level performance measurement frameworks such as under the Chemicals Management Plan or the Clean Air Regulatory Agenda.

Scorecard - Healthy Environments and Consumer Safety Branch
Criterion Rating Conclusion Rec #
Governance
1.1 Governance structures NMI Enhanced scientific research governance model to be implemented in 2012-13. 1
1.2 Planning NMO A scientific research planning exercise would set a clear strategic direction for the programs, support priority setting and subsequent decision-making at the Directorate, program and Branch levels. 2
Risk Management
2.1 Integrating risk into scientific research NMI Research should act as a mitigating strategy to reduce risk to public health and safety. 2
Internal Control
3.1 Scientific research project selection NMI Research template to be fully implemented in 2012-13 leading to improved internal controls related to: project selection practices, alignment of risks, monitoring of costs and timelines. 2
Results and Performance
4.1 Performance measurement NMO Performance information is needed to aid in decision-making to facilitate the planning and management of scientific research; these should be monitored and the results clearly communicated. 3
S
Satisfactory
NMI
Needs Minor Improvement
NMO
Needs Moderate Improvement
NI
Needs Improvement
U
Unsatisfactory
UKN
Unknown; Cannot Be Measured

Part II - Health Products and Food Branch

The scientific research in this Branch is divided between two key program activities: health products; and food safety and nutrition. The Branch reports expenditures for 2011-12 of approximately $13.7 million for scientific research.

Research Projects by Directorate
Percentage of Total Project Count

Biologics and Genetic Therapies Directorate: 18%, Food Directorate: 82%

The Food Directorate has three bureaus that conduct scientific research: Bureau of Microbial Hazards; Bureau of Nutritional Sciences; and Bureau of Chemical Safety. For fiscal year 2011-12 the Branch had approximately 72 research projects underway. The Regions and Programs Bureau supports the Food Directorate by conducting portions of certain food research projects. Operations and maintenance funding for this research is received through branch transfer agreements. Last fiscal year, the regional laboratories participated on 12 scientific research projects related to food.

The Biologics and Genetic Therapies Directorate is divided between three centers: Centre for Vaccine Evaluation; Centre for Evaluation of Radiopharmaceuticals and Biotherapeutics; and the Centre for Blood and Tissues Evaluation. However, scientific research is only conducted in the Centre for Vaccine Evaluation. The Centre had approximately 16 research projects underway last fiscal year.

1. Governance

1.1 Governance structures

Audit criterion: Governance structures should set priorities and integrate scientific research into branch decision-making processes.

Governance decisions should play a fundamental role in strategically directing scientific research. It would be expected that scientific research priorities would be tabled for decision at a senior committee and that results from the research would be integrated into the Branch decision-making process.

The Health Products and Food Branch has its Branch Executive Committee which is its senior most decision-making body, chaired by the Assistant Deputy Minister and is composed of senior branch management. This Committee is supported by three sub-committees: the Branch Operations Committee, Policy Directors Committee, and International Committee. In addition, the Program Executive Committee (PEC), along with its own sub-committees, is currently in place. As well, the Branch receives scientific advice from advisory groups and expert advisory committees on specific topics related to their expertise. For example, the Biologics and Genetic Therapies Directorate is supported by two expert advisory committees - one on Blood Regulations and the second provides advice on Cells, Tissues and Organs. The committees typically meet two times a year. Likewise, the Food Directorate is supported by two advisory committees - one related to dietary sodium reduction and another specific to food regulations.

At the directorate level where research projects are identified and initiated the management teams are informed of the scientific research project details. While there is a Branch Executive Committee and sub-committees, it was not clear if scientific research priority setting; resource allocation; management of expenditures; and performance measurement for scientific research occurs in a formal and documented way.

Some actions are currently in progress at Health Products and Food Branch to enhance existing governance to encompass scientific research to guide decision-making.

Recommendation 1

It is recommended that the Assistant Deputy Minister, Health Products and Food Branch implement an enhanced branch level governance for scientific research supported by the program governance.

Management response

Management agrees with this recommendation.

Scientific research is a program matter and as a result discussions related to scientific research will occur at the Program Executive Committee (PEC) sub-committee level and will report to the Program Executive Committee (PEC).

As Health Products and Food Branch scientific research is linked to specific programs (for example, food/nutrition and biologics), the management framework will occur at the appropriate Program Executive Committee (PEC) sub-committee level after going through a Directorate level process first.

As well, Health Products and Food Branch has recently created a Branch Science Network with supporting terms of reference to enhance its governance of scientific research.

1.2 Planning

Audit criterion: The branch should have strategic and operational plans for scientific research.

Setting out a strategic direction for scientific research should assist branches in reaching their operational research goals and strengthen inter-branch partnerships between those who conduct research and those who implement its findings. By integrating scientific research into the planning exercise this may be useful in supporting: priority setting; allocation of resources for projects (funding and staff); and reporting.

The Health Products and Food Branch has developed its 2012-15 Strategic Plan which is based on three pillars: a modernized regulatory framework; operational excellence and a people agenda. The Strategic Plan does not detail the role of scientific research; however scientific research is fundamental to the work of the Branch. Integrating scientific research into the planning exercise would support achieving the strategic direction of the Branch and support priority setting and subsequent decision-making at the Directorate level concerning research project selection.

The Branch conducts its operational planning using the Integrated Planning and Performance Reporting System (IPPRS), which promotes uniformity in planning. Branches collect data in the IPPRS in a single coordinated process where they are to identify activities as well as supporting resources. In addition, the Food Directorate manages its research projects in the Program Management Reporting System. One of the reporting areas of the system is research and development. This allows the directorate to track research projects with a specific internal order number which facilitates reporting on key Treasury Board submissions such as the Food and Consumer Safety Action Plan, Food Fortification and Listeriosis.

The Health Products and Food Branch did provide a comprehensive list of research projects for both the Food Directorate and the Biologics and Genetic Therapies Directorate. While being able to compile a list of the scientific research projects is an important first step, in itself, it does not constitute planning. The Directorate notes that more accurate information on staff time applied against the specific research projects would be beneficial. As well, information on salary usage by project is not available from the departmental financial system (SAP) or from the departmental planning system. The financial system can only provide information on actual operating and maintenance and capital usage costs.

The Food Directorate uses the Project Management and Reporting System as a repository of all its research projects. Each research project is documented with an abstract, detailing how it will contribute to the mandate, specific linkages to the Branch priorities, goals, and start and finish dates. However the project reporting system is missing source of funding information, expenditures information, and research results. This information is required to facilitate the management of research expenditures.

In examining the costs associated with many projects within these systems, costs could not be reconciled with project costs found in Health Canada's financial system. As a result, individual projects reported differences in expenditures. Numerous projects showed expenditures in excess of plans. As well, there were a large number of questionable expenditures assigned to research projects. The audit team also noted that, in many cases, cost centre information was incomplete and outdated. For example, a number of cost centres did not have an identified manager or reflected outdated management information. Although some directorates have seen a change in their director general, these changes are not reflected in the cost centres. Finally, directorates that no longer exist are still listed as active.

Annually, Health Canada reports to on the Federal Science Expenditures and Personnel Survey (CANSIM 358-0142 to 358-0151). The Survey collects data which are essential to assure the availability of pertinent statistical information to identify expenditures on research and experimental development; to monitor science and technology related activities in Canada; and to support the development of science and technology policy. The data collected also form part of the Gross Domestic Expenditures on Research and Development. However, through interviews, it was determined that the definition of scientific research is widely interpreted. This was confirmed following a detailed financial analysis, which showed a variance between the audited expenditures and those reported for the purposes of the Survey. The Branch should agree upon the interpretation of what constitutes scientific research in order to facilitate more accurate and reliable annual reporting.

The management of scientific research can be strengthened by establishing an agreed upon definition for scientific research to support planning, the allocation of the resources, and the more accurate and reliable reporting of expenditures. This will help the Branch to secure a scientific research portfolio that demonstrates the most effective investment of research dollars as well as a portfolio that is responsive to emerging and important public health needs.

Recommendation 2

It is recommended that the Assistant Deputy Minister, Health Products and Food Branch build on existing practices to:

  • define the spectrum of scientific research;
  • detail scientific research plans; and
  • monitor and report expenditures and personnel devoted to scientific research.

Management response

Management agrees with this recommendation.

  1. Internal (Branch) discussion and agreement on a unified definition for scientific research for the purpose of planning and reporting.
  2. Programs will enhance the communication of their scientific research priorities and plans. Scientific research plans will be presented to Partnership Executive Committee at the appropriate time in the planning process.
  3. Health Products and Food Branch will seek to integrate research into the planning process given the limitations of the current planning systems. Through the implementation of the standardized activity structure, revamped financial activity codes and SAP project systems across the Branch, Health Products and Food Branch will capture the costs of the research being conducted and will include this as part of the program planning discussion. Reporting of expenditures and personnel devoted to scientific research will be reported through the Program Executive Committee at the appropriate times in the planning and reporting cycle.

2. Risk management

2.1 Integrating risk into scientific research

Audit criterion: Branch program risks should direct scientific research activities.

Scientific research should contribute to greater knowledge towards mitigating Branch program risks. Regular review of scientific research and its results should inform the risk process. As such, research objectives should be linked to reducing identified health science risks.

The Health Products and Food Branch has developed a high level registry relating to its Branch risks. An analysis of the Food Directorate research projects for 2010-11 was completed and noted that many research projects are well described with titles, abstracts, themes, risk analyses, work plans and timelines. However, there was some inconsistency on the quality of the risk information input into the system.

Over the course of the audit, the governance structure for the management or scientific research has evolved. Organizationally, the Branch's Risk Management and Decision Sciences Division is now located within the Policy, Planning and International Affairs Directorate. This re-organization has the potential to centralize the review of Branch scientific research results towards supporting the regulatory programs through risk mitigation analysis. (See recommendation 2)

3. Project management in support of scientific research

3.1 Scientific research project selection

Audit criterion: Research projects should be aligned with the branch mandate and with established criteria for project selection.

Scientific research project selection and the associated funding should be the "official record" of what was decided by senior management. Annual Branch operational planning should be used to guide the research selection process and record the financial amounts committed throughout the life cycle of the research project.

The Health Products and Food Branch uses project and program peer review as standard tools in the evaluation of scientific research. There was little documentation of the processes undertaken and associated reporting to Branch management. Although the Branch indicated that the prioritization of research projects occurs during the annual operational planning process and is guided by Branch priorities, there was little documented evidence. Joint ventures and Treasury Board submissions have more up-front clarity in project selection. (See Recommendation 2)

4. Performance measurement

4.1 Performance measurement

Audit criterion: Scientific research should have performance measures which are monitored and reported.

Branch strategic direction for its scientific research should be supported by the research work performed and results obtained. A performance measurement framework for scientific research is an integral part of managing for results.

The documentation examined shows that the Branch makes efforts to communicate science results so that knowledge is passed on and results are understood. The Branch also recognizes that, to ensure transparency, it must articulate its scientific activities in a way that a lay audience can understand. Effective science communication will result in increased partnerships and support and may even inspire a young generation of scientists or students in scientific disciplines.

The audit notes that performance information focused on research project activities and outputs, which informs management's decision-making at the operational level. Additional information to inform strategic decision-making, planning, direction setting and performance reporting would be important. This includes the need for information on the status of key activities and the identification of significant issues or delays and the tracking of progress in key outputs and outcomes. Evidence that demonstrated progress reporting on high profile projects which included key performance indicators, targets, progress attained, as well as risks / mitigation strategies was collected and examined. However this was not consistent across all scientific research projects. It would be beneficial for the Branch and its directorates to employ a similar approach for other scientific research projects.

As stated previously, the Food Directorate uses a Project Management and Reporting System as a repository of all its research projects. Within this system, in some instances, evidence was found for performance measures. However, there was no evidence of reports on progress based on these measures. The Health Products and Food Branch would benefit from analyzing progress and reporting against performance measures for its scientific research projects, and identifying areas for potential improvement.

As noted, the 2012-15 Strategic Plan, will direct the Branch's work to support three pillars: plans a modernized regulatory framework; operational excellence and a people agenda which should strengthen planning and reporting and use the best science to inform decisions. The Branch, through this process, aims to ensure the priorities remain relevant and effective by developing annual work plans and continuing to review the Plan and reporting on priorities through the operational planning and reporting processes incorporated in the Department's planning and performance reporting system.

Recommendation 3

It is recommended that the Assistant Deputy Minister, Health Products and Food Branch extend performance measures for all its scientific research.

Management response

Managementagrees with this recommendation.

The Health Products and Food Branchwill develop performance reporting for scientific research. Research metrics, including costs and budget will be discussed as part of the program approach through Partnership Executive Committee.

Scorecard - Health Products and Food Branch
Criterion Rating Conclusion Rec #
Governance
1.1 Governance structures NMO Enhance the governance model to better integrate the scientific research activity. 1
1.2 Planning NMI Food Directorate uses the Project Management Reporting System for the purposes of planning. The Biologics, Genetics and Therapies Directorate (BGTD) does not have a similar system for planning but has a lower volume of projects. 2
Risk Management
2.1 Integrating risk into scientific research NMI Food Directorate uses the Project Management Reporting System to populate specific information related to the individual scientific research projects which has a section for risk analysis. Most were well completed. BGTD could benefit from a similar process. 2
Internal Control
3.1 Scientific research project selection NMO Improved project selection practices and criteria will allow the Branch to better link the research projects to program objectives and activity. 2
Results and Performance
4.1 Performance measurement NMO The Food Directorate has some performance information documented in individual research projects; the Project Management Reporting System expects performance information and timelines to be completed. The Centre for Vaccine Evaluation does not use the system for its 16 projects. 3
S
Satisfactory
NMI
Needs Minor Improvement
NMO
Needs Moderate Improvement
NI
Needs Improvement
U
Unsatisfactory
UKN
Unknown; Cannot Be Measured

Part III - Pest Management Regulatory Agency

The Pest Management Regulatory Agency (PMRA) is the regulatory authority for pesticides. The Branch's mandate is to prevent unacceptable risks to human health and the environment through the regulation of pest controls products. Under the Pest Control Products Act, the PMRA: registers pest control products for use in Canada; develops pest management policies and guidelines; promotes sustainable pest management; looks to improve the regulatory process to increase efficiencies; and distributes pest management information to the general public and key stakeholders. The PMRA is responsible for assessing the risks associated with new pest control products in relation to human health and the environment, conducting value assessments on pest control products, re-evaluating older pest control products against modern scientific standards, and carrying out compliance and enforcement activities.

PMRA Research Needs
Percentage of Total Project Count

Environmental Assessment Directorate: 40%, Health Evaluation Directorate: 48%, Value Assessment and Re-Evaluation Management directorates: 12%

The PMRA relies on research and monitoring produced by other federal departments and agencies with research, policy or regulatory capacity related to pesticides and/or pest management, known as the 6 Natural Resources Departments and Agencies. The PMRA has a research plan identifying research and monitoring priorities. While the plan outlines the Agency's research and monitoring priorities, the realization of the research priorities depends on delivery from the 6 Natural Resources Departments and Agency.

For fiscal year 2011-12 the Branch identified over 20 scientific research needs to be conducted within this partner arrangement. Approximately 10 of the research projects outlined in the plan fit the audit definition of scientific research. Of the 10, one project is currently being conducted by Health Canada's Healthy Environments and Consumer Safety Branch, to which the PMRA has provided input into the study design.

While the PMRA's laboratory is primarily used for regulatory testing, it does conduct a small amount of scientific research. The laboratory has a primary mandate to support the pest management regulatory system through the National Pesticide Compliance Program by providing analytical testing services (pesticides) and scientific advice. According to the Agency's management, 95 percent of the laboratory analytical testing is done using standard or published methods. In the absence of published analytical methods, in-house methods are developed or adapted by the laboratory which constitutes scientific research as defined by Health Canada management.

1. Governance

1.1 Governance structures

Audit criterion: Governance structures should set priorities and integrate scientific research into branch decision-making processes.

Governance decisions should play a fundamental role in strategically directing scientific research. It would be expected that scientific research priorities would be tabled for decision at a senior committee and that results from the research would be integrated into the Branch decision-making process.

The PMRA's governance structure consists of three main committees: the Agency Management Committee, the Science Management Committee and Science Operations Committee. The Agency Management Committee is the senior management decision-making committee for all policy and operational matters. The Science Management Committee, among its many roles, discusses and decides on areas of significant importance to PMRA, including, but not limited to, science, pesticide registrations, process management, policy, and risk management. The Science Operations Committee reviews and discusses scientific data and issues as they relate to submissions for pesticide registrations, and make recommendations to the Science Management Committee on submission. The identification of the PMRA research needs is reviewed and recommended for approval by the Science Management Committee. Subsequent approval is granted by the Agency Management Committee.

A third key committee is the Interdepartmental Working Group on Pesticides and Pest Management. As previously noted, to support its mandated activities, the PMRA relies on research and monitoring produced by other federal departments and agencies with research, policy or regulatory capacity related to pesticides and/or pest management, known as the 6 Natural Resources Departments and Agency. The partnership is guided by a Memorandum of Understanding between Health Canada, Agriculture and Agri-food Canada, the Canadian Food Inspection Agency, Environment Canada, Fisheries and Oceans Canada, and Natural Resources Canada. Annually, these six member departments and agencies conduct a priority setting exercise and produce an integrated work plan. It is the responsibility of each member department and agency to implement their respective work plan responsibilities.

1.2 Planning for scientific research

Audit criterion: The Agency should have strategic and operational plans for scientific research.

Setting out a strategic direction for scientific research should assist branches in reaching their operational research goals and in strengthening inter-departmental partnerships between those who conduct research and those who implement its findings. By integrating scientific research into the planning exercise this may be useful in confirming the strategic direction, support priority setting and subsequent decision-making at both the Directorate and Branch levels concerning research project selection.

Research Priorities by Theme

  • Toxicology
  • Exposure (General)
  • Dietary Exposure
  • Occupational/ Residential/ Bystander Exposure
  • Risk Assessment
  • Population Studies/Surveys Biological Monitoring

The PMRA has a research plan identifying research and monitoring priorities. The research plan identifies pesticide and pest management themes (such as toxicology), research partners, any external research activity, and for each need there is a priority rating (high, medium or low). The PMRA reports that while they have a documented plan outlining its research and monitoring priorities, the realization of the plan depends on delivery from the other natural resources departments and agency.

In February 2011, the Executive Committee - Finance, Evaluation and Accountability approved the summative evaluation "Building Public Confidence in Pesticide Regulation and Improving Access to Pest Management Products - Horizontal Initiative." In the evaluation there was a recommendation to develop a joint strategy to maintain and strengthen the focus of the pesticide-related research and monitoring work in support of the PMRA's priority needs. In the PMRA's management action plan response to the summative report they committed to developing a strategy to strengthen the focus of pesticide-related research and monitoring work by March 2012. The strategy, which includes an integrated work plan, has been developed in collaboration with the other natural resources organizations.

2. Risk management

2.1 Integrating risk into scientific research

Audit criterion: The Agency program risks should direct scientific research activities.

Scientific research should contribute to greater knowledge towards mitigating Branch program risks. Regular review of scientific research and its results should inform the risk process. As such, research objectives should be linked to reducing health science risks identified.

Through interviews the PMRA noted that the availability of pesticide and/or pest management research and monitoring data, from working group organizations, had improved the quality of risk assessments and supported decision-making. They also noted that collaboration among federal departments and agencies had greatly improved the identification of research priorities and the flow of scientific expertise. However, as mentioned, it is the decision of each member department and agency to conduct specific work plan responsibilities as budgets and other priorities allow. Shifting priorities within the 6 Natural Resources Departments and Agency has had an impact in the past on the completion of the research and monitoring integrated work plan. The recently signed integrated work plan re-confirms priorities and notes that the approach provides opportunities for all partners to work more effectively by identifying common areas of research allowing for a multi-departmental participation and the more efficient use of resources.

3. Project management in support of scientific research

3.1 Scientific research project selection

Within the PMRA, the responsibility for coordinating identified pesticide research and monitoring activities rests with the Director General working group representing the 6 Natural Resources Departments and Agency. This working group considers the priorities presented to it by the PMRA and updates its integrated work plan as required and as budgets allow. The PMRA identifies research and monitoring priorities in partnership with stakeholders and international regulatory agencies. For example, workshops have been held with producers, provincial/federal government representatives, international partners, and pesticide industry representatives in attendance. Joint priorities are also identified with international partners. A recent survey found that consultations with stakeholders to set priorities were highly or somewhat effective with over 70 percent of respondents.

In discussion with staff, it was noted that while the PMRA can request research to be performed by the working group, the member departments and agencies must prioritize Health Canada requests within their own priorities. As noted, annually, the PMRA identifies its research needs but also includes the status of these requests. There is a risk that its partner organizations may not completely meet the PMRA's research needs. Consequently, the Agency should continue to identify and document alternative approaches to address potential shortfalls in pesticide and/or pest management research and monitoring.

4. Performance measurement

4.1 Performance measurement

A performance measurement framework for scientific research is an integral part of managing for results. The PMRA's strategic direction for its scientific research is supported by the research requests and results obtained. Research results and monitoring data provide the PMRA with better information on the presence and effects of pesticides in the environment and food supply, improve the body of knowledge on pesticides, and contribute to decisions that are protective of human health and the environment. As well, the PMRA works in collaboration with international colleagues in monitoring pesticide research and in sharing scientific expertise.

In 2007, the PMRA began producing an annual report on pesticide incidentsto improve the capacity to monitor health and environmental incidents related to pesticide use and exposure and to help the PMRA develop corrective actions when necessary. Registrants are required by law to report incidents involving their products to the PMRA. The incident reporting database is used by scientific evaluators to look for trends and additional information about pesticides to support the risk assessment process. In 2008, the voluntary program for the public to report incidents was updated, and an electronic reporting form for registrants was updated to improve data collection. Over the coming years, incident reports will play an increasingly important role in improving the risk assessment process, in improving product label information, and in the development of prevention and education programs.

The PMRA participates in a peer review process to evaluate ongoing funded research activities which positions them well to evaluate the results obtained. It also shares with the other participants' respective priorities for pesticides research and monitoring activities and monitoring activities that focus on regulatory needs.

The 2011 summative evaluation had positive findings related to the research and monitoring stream and noted the good horizontal coordination in support of science-based regulatory decision-making.

Scorecard - Pest Management Regulatory Agency
Criterion Rating Conclusion Rec #
Governance
1.1 Governance structures S The Pest Management Review Agency's governance related to its research requests is guided by a Memorandum of Understanding, interdepartmental committee and departmental committee. Evidence of agenda setting and strategic direction was evident.  
1.2 Planning S Although there is a research plan identifying research and monitoring priorities, the plan is fully dependent on delivery from the other departments and agency.  
Risk Management
2.1 Integrating risk into scientific research S Each member department and agency must prioritize the PMRA's priorities within their own scientific research priorities. Shifting priorities within these departments have an impact on the completion of the research and monitoring of the integrated work plan.  
Internal Control
3.1 Scientific research project selection n/a    
Results and Performance
4.1 Performance measurement n/a    
S
Satisfactory
NMI
Needs Minor Improvement
NMO
Needs Moderate Improvement
NI
Needs Improvement
U
Unsatisfactory
UKN
Unknown; Cannot Be Measured

Appendix A - Lines of enquiry and audit criteria

Line of Enquiry 1: Governance
1.1 Governance structures Governance structures should set priorities and integrate scientific research into branch decision-making processes.
1.2 Planning The branch/Agency should have strategic and operational plans for scientific research.
Line of Enquiry 2: Risk Management
2.1 Integrating risk into scientific research Branch/Agency program risks should direct scientific research activities.
Line of Enquiry 3: Project Management in Support of Scientific Research
3.1 Scientific research project selection Research projects should be aligned with the branch mandate and with established criteria for project selection.
Line of Enquiry 4: Performance Measurement
4.1 Performance measurement Scientific research should have performance measures which are monitored and reported.

Appendix B - Overall scorecard

Criterion Ratings Conclusion Rec #
HECSB HPFB PMRA
Governance
1.1 Governance structures NMI NMO S Enhance the governance model for the Healthy Environments and Consumer Safety Branch (HECSB) and the Health Products and Food Branch (HPFB) to better integrate the scientific research activity. 1
1.2 Planning NMO NMI S For HPFB and HECSB better overall financial management for the allocation of resources and final costing of projects. 2
Risk management
2.1 Integrating risk into scientific research NMI NMI S Research-related risks mostly identified and prioritized in project abstracts. 2
Internal Control
3.1 Scientific research project selection NMI NMO n/a Improved project selection practices and criteria to be used during the operational planning process will allow the branches to better link the research projects to program objectives. 2
Results and Performance
4.1 Performance measurement NMO NMO n/a Performance information is needed to aid in decision-making to facilitate the planning and management of scientific research; these should be monitored and the results communicated. 3
S
Satisfactory
NMI
Needs Minor Improvement
NMO
Needs Moderate Improvement
NI
Needs Improvement
U
Unsatisfactory
UKN
Unknown; Cannot Be Measured
Report a problem or mistake on this page
Please select all that apply:

Thank you for your help!

You will not receive a reply. For enquiries, contact us.

Date modified: