Audit of the Science and Technology Branch’s Laboratory Activities

The Audit and Evaluation Branch
June 2014

Table of Contents

• Key Dates
• Glossary/List of Acronyms
• Acknowledgments
• Executive Summary
• 1 Introduction
  • 1.1 Background
  • 1.2 Objective and Scope
  • 1.3 Statement of Conformance
• 2 Findings
  • 2.1 Implementation of the science plan
  • 2.2 Communication of Science and Technology (S&T) Branch’s plans and activities
  • 2.3 Dissemination of Research and Development (R&D)/Related Scientific Activities (RSA) results
  • 2.4 Laboratory activities and operational planning
  • 2.5 Lessons learned exercises
  • 2.6 Monitoring and reporting laboratory activities
  • 2.7 Management of R&D/RSA work
• 3 Recommendations
• 4 Conclusion
• Annex 1 - Audit Methodology and Criteria

Key Dates

Glossary/List of Acronyms

ADM

Assistant Deputy Minister (S&T Branch)

CESD

Commissioner of the Environment and Sustainable Development

EC

Environment Canada

FTE

Full-Time Equivalents

RSA

Related Scientific Activities

R&D

Research and Development

S&T

Science and Technology

CALA

Canadian Association of Laboratory Accreditation

LIMS

Laboratory Information Management System

Prepared by the Audit and Evaluation Team

Acknowledgments

The audit team comprised of Jean-Luc Tétreault, audit manager, and Philippe Knight and Daniel Chénier, auditors, under the direction of Stella Line Cousineau, would like to thank those individuals who contributed to this project and, particularly, employees who provided insights and comments as part of this audit.

File version

File: B.6.4 - English - Final Report STB’s Laboratory Activities approved by DM
Date: June 13, 2014

Executive Summary

This audit was included in the departmental 2012 Risk-Based Audit Plan approved by the Deputy Minister, upon recommendation of the External Audit Advisory Committee. The purpose of the audit was to determine whether the Science and Technology (S&T) Branch has effectively managed its laboratory activities.

Overall, the audit found that EC’s S&T Branch laboratory activities were generally well managed. For instance EC’s science plan is clearly articulated and both management and staff understand the part they play in the plan. Laboratory activities and operational plans are well aligned with S&T priorities. Practices are in place to ensure the timely dissemination of laboratory activities results. As well, there are adequate laboratory monitoring processes, and monitoring reports provide regular information on the progress of operational and research laboratory work.

However some opportunities for improvement were also identified such as the process for the S&T Branch directorates’ long-term planning and better communication to staff in the areas of planning and lessons learned. In addition, ensuring that key managerial functions (such as planning and other administrative matters) are consistently and effectively conducted would improve overall management of laboratory activities.

To address the findings outlined in this report, we present the following recommendations.

The Assistant Deputy Minister, Science and Technology should:

1. Determine how the current S&T Branch directorates’ long-term planning process can be improved by building on existing best practices.
2. Implement regular communication between management and staff regarding S&T Branch’s laboratory activities, their relationship to departmental priorities and the needs of departmental users, and lessons learned.
3. Review the laboratory managerial structure to improve the consistency and effectiveness of managerial and administrative support throughout S&T Branch laboratories.

Management agrees with the recommendations. The detailed management response can be found under Section 3 of this report.

1 Introduction

This audit was included in the departmental 2012 Risk-Based Audit Plan approved by the Deputy Minister, upon recommendation of the External Audit Advisory Committee.

1.1 Background

Under the Canadian Environmental Protection Act, 1999 and the Department of the Environment Act, Environment Canada (EC) is mandated and enabled to conduct scientific research and environmental monitoring in order to provide robust scientific data, knowledge and evidence to support policy, regulations, enforcement, and decision-making. For example, data generated by EC’s scientific activities provides the evidence required for enforcement action under the Canadian Environmental Protection Act, 1999, the Fisheries Act, the Migratory Bird Convention Act and Species at Risk Act (DNA analysis), as well as the Management of Contaminated Fisheries Regulations by the Department of Fisheries and Oceans.

Through its laboratory activities EC makes use of technology to support regulatory initiatives and develops methods and devices to support regulation development and enforcement. As a result, EC’s laboratory work includes technologies that are critical to meeting the Department’s legislated obligations.

Science and Technology (S&T) plays a significant part in EC’s efforts to carry out the Department’s mandate, representing over half of EC’s budget and staff. The Department’s scientific expertise and interests cover a wide range of areas - atmospheric, water and wildlife.

According to international convention, S&T activities are divided into research and development (R&D) and related scientific activities (RSA). Together, R&D and RSA are the backbone of EC’s scientific capacity, providing basic and applied knowledge necessary to fulfill the Department’s policy, regulatory and enforcement mandates effectively and in a manner that maintains public trust. R&D activities largely focus on the production of new scientific knowledge (e.g. find new ways to measure the level of toxic substances in the environment). An R&D project normally has three characteristics:

• A substantial element of uncertainty, novelty and innovation;
• A well-defined project design; and
• A report on the procedures and results of the project.

RSA mostly focus on the application of existing scientific knowledge (e.g. perform environmental monitoring activity based on generally accepted toxicity measurement methods). Staff in EC’s laboratories support and/or perform both R&D and RSA work. RSA complement and are an extension of R&D, by contributing to the generation, dissemination and application of scientific and technological knowledge.

For the 2011-12 fiscal year, EC spent a total of $653 million on S&T activities, including $238 million on R&D and the remaining $416 million on RSA. The Department also allocated for the same period a total of 3,353 full-time equivalents (FTEs) on S&T, including 931 FTEs on R&D and 2,422 FTEs on RSA.^Footnote 1Only a portion of these resources are being currently used for the delivery of laboratory activities.

To ensure that R&D and RSA work addresses environmental issues that are of the highest priority for EC and the Government of Canada, the Department has set out its long-term S&T priorities in the 2007 Science Plan and the 2010 Technology Role. In the context of this audit report the Science Plan and Technology Role documents are together referred to as EC’s science plan. Building on this previous work, at the time of this audit EC was developing a new Science Strategy, which will set new long-term priorities and will focus on the impact and contribution of S&T activities to EC’s policies, programs and services. The new Science Strategy will be driven by five key principles: relevance, collaboration, responsiveness, transparency and excellence.

Most of EC’s R&D and RSA laboratory work is conducted through the S&T Branch and has the objective of providing information leading to a better understanding of the various aspects of our environment (e.g. air, water, wildlife, biodiversity, soil and climate). In turn, this information supports the Department’s ability to deliver its mandate, including its commitment to be a world class regulator, its enforcement activities, its weather services and its policy development functions. S&T Branch work is carried-out in 21 research centres, located across Canada, with more than half outside the National Capital Region. Many of the research centres have laboratories. Key directorates under the S&T Branch engaged in R&D/RSA laboratory work are the:

• Atmospheric Science and Technology Directorate;
• Water Science and Technology Directorate; and
• Wildlife and Landscape Science Directorate.

In December 2011, the Commissioner of the Environment and Sustainable Development (CESD) conducted a performance audit on “Environmental Science.” Overall, the audit concluded that the Department had developed a Science Plan that sets out long-term commitments for undertaking, managing, and applying its science for the next 10 years. However, the CESD audit noted that the Department had yet to implement the Plan or integrate it into its overall results management framework. In addition, the audit concluded EC has not established an operational plan necessary to achieve the strategic priorities set out in its Science Plan or to measure progress against those priorities.

The CESD recommended integrating the Plan in the Department’s results management framework, developing objectives and targets for the Plan and integrating it in the overall planning process of the Department.

Audit and Evaluation Branch is following up on the progress of these recommendations, in the present audit and via the ongoing mechanism of quarterly reports. EC staff told us that as part of launching the new Science Strategy, an implementation plan will be presented to EC’s Business Integration Committee that will outline objectives and targets, as suggested by the CESD.

1.2 Objective and Scope

Objective

The main objective of the audit was to determine whether the Science and Technology (S&T) Branch has effectively managed its laboratory activities.

Appendix A of this audit report summarizes the audit methodology and criteria.

Scope

For the purposes of the audit, laboratory activities were defined as any R&D/RSA laboratory work conducted for operational, monitoring and research purposes, other than S&T Branch’s office based R&D/RSA (i.e. modelling and data analysis and interpretation).

The focus of the audit included the full management cycle of R&D/RSA laboratory work, including the planning, conducting, reporting and lessons learned phases of R&D/RSA which used EC’s laboratory services. While the primary focus of the audit is on laboratory activities, some topics examined have broader S&T implications.

The scope of the audit also included a follow-up of the implementation status of the recommendations included in the CESD December 2011 report on “Environmental Science.”

The scope of the audit did not include:

• The management of EC’s laboratories facilities and equipment, since this was considered in the context of an internal audit on Capital Assets Management; and
• Department’s laboratory security and occupational health and safety, as these components have been subject to an audit in 2010-11 for physical security and in 2009-10 for occupational health and safety.

The audit covered primarily the period between April 1, 2012 and March 31, 2013.

1.3 Statement of Conformance

This audit conforms with the Internal Auditing Standards for the Government of Canada as supported by the results of the quality assurance and improvement program.

In our professional judgement, sufficient and appropriate audit procedures have been conducted and evidence gathered, to support the accuracy of the conclusions reached and contained in this report. The conclusions were based on a comparison of the situations, as they existed at the end of the fieldwork in November 2013, against the audit criteria.

2 Findings

Overall the audit found that Environment Canada’s S&T Branch is in general effectively managing its laboratory activities. However, we found opportunities for improvement as described in this section. Also, we noted that the S&T Branch has made progress for its laboratory activities in implementing the recommendations of the 2011 CESD audit.

2.1 Implementation of the science plan

We expected that the leadership of the Branch has clearly articulated the plan for science at EC, that staff understand their role in it, and that the plan is founded on a process to share ideas, involving staff and management.

EC’s science plan is articulated in the 2007 Science Plan and the 2010 Technology Role. The audit noted that the plan is communicated through the S&T Branch Assistant Deputy Minister’s message to staff outlining key departmental S&T priorities, and through S&T Branch directorates’ long-term planning that outlines each directorate’s specific priorities. A review of these documents confirmed that S&T Branch’s laboratory activities are well aligned to EC’s S&T priorities.

The Branch’s process for the sharing of ideas between staff and management specific to EC’s science plan is reflected in the development of the directorates’ long term planning documents. These documents incorporate EC’s priorities and demonstrate that they were developed using a bottom-up and top-down approach, and a participative methodology.

S&T Branch directorates also support the implementation of the plan by developing and providing periodic planning presentations to their staff on “what the Department’s S&T priorities mean for them.” A review of some of these presentations of planning documents and an assessment of staff satisfaction in that area indicated that management's communication of the plan and priorities was effective.

The audit noted a best practice in the Atmospheric S&T Directorate. This area has an effective and systematic process for its long-term planning (e.g. foresight), resulting in science projects more clearly aligned to the plan and more forward looking. This process also increases engagement of staff and management in implementing EC’s science plan. The audit noted that other directorates within S&T Branch could benefit from adopting a similar approach to planning.

2.2 Communication of S&T Branch’s plans and activities

The attribute of a “well managed research organization” stresses the importance of having systems and practices in place for communicating its plans and activities

In order to provide the support and communicate the alignment of plans to EC’s science plan, periodic management-staff formal and informal meetings are being conducted. We reviewed the agenda, minutes or records of decision for the formal meetings, including presentations delivered by management, and we interviewed staff on their satisfaction in that area. The audit found that periodic management-staff meetings took place. We found a good practice in the Air Quality Research Division where one manager periodically reviews and communicates the meaning of the departmental priorities with his staff, the resourcing strengths and weaknesses and administrative challenges that may have an impact on R&D/RSA (e.g. staffing, procurement and information management/information technology). This practice could be adopted more broadly and be beneficial to other directorates within S&T Branch.

The science plan is also implemented and communicated through the development of the S&T Branch directorates’ laboratory activities work plans and research proposals. A review of these documents indicated that they were aligned to EC’s science plan, and related interviews demonstrated that staff understood the role of Branch plans and activities in EC’s science plan.

In addition, management conducts regular meetings formally or informally with staff to provide updates and discuss operational issues related to the EC science plan. We noted that processes were in place to support these discussions and found staff to be generally satisfied with the current communication approach, and found these regular meetings to be both effective and timely.

Resistance to change within S&T Branch in implementing the science plan was not identified as an issue. Therefore, we did not pursue additional audit work in that area.

2.3 Dissemination of R&D/RSA results

Effective dissemination of R&D/RSA laboratory results implies these are delivered in a manner which is both timely and responsive. We expected to find a process that takes into consideration laboratory activity requirements from all users and allows for the timely dissemination of results. The process should be able to adapt to changes related to the users’ needs.

Through the testing of a statistical sample of laboratory activities (Laboratory Information Management System (LIMS) and non-LIMS) the audit concluded that practices to ensure timely dissemination of R&D/RSA results existed and were effective. In addition, a client survey process is currently in place for laboratory activities driven by LIMS and results are shared with management for decision making. The survey captures information such as whether the delivery of laboratory results was done in a timely and responsive manner, whether they have received the right service quality (etc.).

Results of the testing and interviews also determined that R&D/RSA are generally adapting to the needs of departmental users and that processes are in place to communicate those changes within S&T Branch directorates. Specifically, we noted that one of these communication processes was found to be working better than others because the process was more future oriented, which can lead to better relationships between users and providers, and is often a success factor in ensuring that users’ needs are satisfied.

2.4 Laboratory activities and operational planning

At the research project level, we expected to find that the need for laboratory activities is periodically assessed in relation to the departmental mandate, for relevance purposes, and that appropriate operational plans are developed and implemented.

The auditors assessed the relevance of the R&D/RSA work by comparing the rationale for the work conducted at the project level to EC’s S&T priorities. A sample of LIMS and non-LIMS projects was included in this analysis. The audit concluded that laboratory activities were well aligned with EC’s S&T priorities and mandate.

In addition, the audit examined whether operational plans “at the project level” were developed and periodically updated by EC staff. Operational plans are a means to provide a common understanding of what needs to occur to implement EC’s S&T priorities. They identify: what specific actions need to be taken, what the results to be achieved are, who is responsible for the results, how the results will be achieved, and what the specific timelines are. The audit concluded that operational plans for the selected R&D/RSA projects existed, were aligned and included the key elements required for an effective operational plan.

While this audit focussed on the operational planning at the research project level, the 2011 CESD audit on “Environmental Science” focussed at a more macro level. As such the CESD noted a lack of operational plans that contained the necessary objectives, targets, and actions to achieve the strategic directions and commitments as included in EC’s 2007 Science Plan.

In the context of the CESD's audit, a follow-up was done to assess the progress made against this outstanding recommendation. Essentially, as part of the new Science Strategy an implementation plan outlining objectives and targets is to be developed and presented to EC’s Business Integration Committee.

2.5 Lessons learned exercises

We expected to find that the laboratory work included processes for:

• Identifying lessons learned;
• Providing access to lessons learned; and
• Using the results of lessons learned.

Identifying lessons learned - The auditors found that some lessons learned exercises are completed by S&T Branch staff. However, these are not being completed on a systematic basis in all of S&T Branch’s directorates.

Providing access to lessons learned - The audit found through a review of selected departmental documentation that results from the conduct of lessons learned exercises are at times reported and accessible to S&T Branch staff and clients/stakeholders. In addition, the audit found that EC’s departmental performance reports occasionally refer to examples of lessons learned related to laboratory activities in their “Lessons Learned at the Strategic Outcome Level” sections, and the same is true for various programs where lessons learned are accessible on specific portals (e.g. Oil Sands). However, a systematic communication of results emanating from these continuous learning exercises is not in place.

Using lessons learned - The audit found that results from the conduct of continuous learning exercises were used for laboratory activities using the Laboratory Information Management System (LIMS). However, the use of lessons learned was not as clearly demonstrated for the non-LIMS laboratory activities. A strategy for the use of results emanating from the conduct of lessons learned exercises can be a key factor to ensure continued progress and improvement.

2.6 Monitoring and reporting laboratory activities

We expected to find processes for monitoring and reporting on the S&T Branch’s laboratory activities.

The audit reviewed the Laboratory Information Management System activity/monitoring reports for EC’s operational laboratories, the Project Tracking Reports for the Wildlife and Landscape Science Directorate and the project tracking/progress monitoring reports (including various planning/reporting templates) for the Water and Atmospheric Science and Technology Directorates. The audit identified that all three directorates have adequate monitoring/reporting processes in place. For example, monitoring reports for operational laboratory work included detailed comparatives of the number of laboratory tests received/reported versus projected on a monthly/yearly basis. The reports for research laboratory work covered various research projects progress monitoring areas - such as the projects’ status on costs/results and deliverables.

2.7 Management of R&D/ RSA work

The audit examined whether effective practices for the management of R&D/RSA were in place. More specifically, the audit assessed that good project management practices as outlined in generally recognized international standards (e.g. goals, resources and accountability for results) existed.^Footnote 2 The audit team reviewed a random sample of laboratory tests and a design test sample of “Research Proposals.” The audit concluded that good project management practices are in place with clear goals, identification of resources required and clear accountability for results.

We also expected to find that the S&T Branch’s staff are performing managerial and administrative support functions consistently throughout the Branch. We found that these functions are performed consistently at the S&T Branch, with the exception of the Water S&T Directorate. In this directorate, as in others, managers play a dual role with both managerial and scientific responsibilities; however, in this case, accountability focuses on their scientific responsibilities and much less on managerial and administrative support responsibilities.

3 Recommendations

To address the findings outlined in this report, we present the following recommendations.

Recommendation 1

The Assistant Deputy Minister, Science and Technology should determine how the current S&T Branch directorates’ long-term planning process can be improved by building on existing best practices. (Section 2.1)

Management Response

Science and Technology Branch concurs with this recommendation.

An Implementation Plan for EC’s new 2014-2019 Science Strategy is being developed and will include mechanisms to incorporate the Science Strategy into the long-term planning of the Branch’s R&D and RSA activities, including laboratory activities.

The Branch will explore current planning processes within each directorate to identify best practices and assess their suitability for adoption in other S&T directorates to improve the effectiveness of long-term planning across the Branch.

Recommendation 2

The Assistant Deputy Minister, Science and Technology should implement regular communication between management and staff regarding S&T Branch’s laboratory activities, their relationship to departmental priorities and the needs of departmental users, and lessons learned. (Sections 2.2, 2.3 and 2.5)

Management Response

Science and Technology Branch concurs with this recommendation.

Several fora for management-staff dialogue have recently been created. The Branch will look at how these fora can be used to further improve the effectiveness of communications between management and staff.

Recommendation 3

The Assistant Deputy Minister, Science and Technology should review the laboratory managerial structure to improve the consistency and effectiveness of managerial and administrative support throughout S&T Branch laboratories. (Section 2.7)

Management Response

Science and Technology Branch accepts the recommendation and acknowledges that Water Science and Technology Directorate (WSTD) is the area in which improvement efforts will be focussed.

In this regard, WSTD has already begun undertaking improvements, including structural changes managerial and administrative support functions.

In particular, a new Finance and Administration (F&A) service model was launched on April 1, 2013. This service-based model seeks to deliver nationally consistent core F&A services within a documented set of standards. The planned implementation of this new model will continue in 2014-2015 and beyond, with a continuous improvement approach.

Specifically with respect to the management of research, WSTD executive approved (October 2013) the implementation of a Research Management structure comprised of three Research Management positions in each research Division, which is similar to the management structure of the other S&T research Directorates. Note that both WSTD research Divisions currently have one Research Manager position.

Staffing has begun and is expected to result in a pool of qualified candidates at the REM 02 level by August 2014. Staffing of the Research Management positions will depend upon the availability of resources.

The implementation of this Research Management structure within WSTD will result in a clear delineation of research and management activities and functions.

4 Conclusion

Overall, the audit found that EC’s S&T Branch laboratory activities were generally well managed. For instance EC’s science plan is clearly articulated and both management and staff understand the part they play in it. Laboratory activities and operational plans are well aligned with S&T priorities. Practices are in place to ensure the timely dissemination of laboratory activities results. As well, there are adequate laboratory monitoring processes, and monitoring reports provide regular information on the progress of operational and research laboratory work.

The audit also identified opportunities for improvement in the areas of S&T Branch directorates’ long-term planning, better communication to staff in the areas of planning and lessons learned and consistent and effective conduct of key managerial functions.

With regards to the 2011 CESD recommendations, the Audit and Evaluation Branch confirms as a result of this audit the status on progress, for its laboratory activities, as conveyed by the S&T Branch in the “Third Quarterly Management Action Follow-Up Summary Report - January 31, 2013” for two recommendations as included in the “Environmental Science” performance audit report. These are: integrate Science Plan in results management framework (2.61); and develop objectives and targets for Science Plan, and integrate Plan in overall planning process (2.69).

Annex 1 - Audit Methodology and Criteria

Methodology

The planning phase for this audit took place from October 2012 to May 2013. A risk assessment was performed to confirm the audit objective and areas that warranted further examination. The audit criteria used in the context of this audit were based on Attributes of Well-Managed Research Organizations developed by the Office of the Auditor General.

The conduct phase for this audit took place at EC headquarters and laboratories located in Moncton (NB), Burlington and Toronto (ON), Montreal (QC), and the National Wildlife Research Center facility located in the National Capital Region (ON) from June 2013 to November 2013.

The audit methodology consisted of:

• Interviews with management and researchers working for the:
  • Atmospheric Science and Technology Directorate (Air Quality Research Division; Climate Research);
  • Water Science and Technology Directorate (Aquatic Contaminants Research Division; Watershed Hydrology and Ecology Research Division; Water Quality Monitoring and Surveillance; Emergencies, Operational Analytical Laboratories and Research Support); and
  • Wildlife and Landscape Science Directorate (Ecotoxicology and Wildlife Health Division).
• Review of relevant departmental documentation (e.g. Canadian Association of Laboratory Accreditation (CALA) assessments, S&T directorates’ strategic and operational information, management’s presentations to staff);
• Review of the document process for a random statistical sample of laboratory tests (25 in the Laboratory Information Management System (LIMS) environment - operational laboratories and Wildlife and Landscape Science Directorate)/ a design test sample for “Research Proposals” (5 from Water Science and Technology and 6 for Atmospheric Science and Technology Directorates); and
• Review of the laboratories activities monitoring and reporting processes/project management practices in each S&T Branch directorate.

Criteria