Evaluation of the Contribution Agreement in support of the Remediation of the Former Mid-Canada Line Radar Sites in Ontario

June 2014

1258-208 (CRS)

Reviewed by CRS in accordance with the Access to Information Act (AIA). Information UNCLASSIFIED.

Acronyms and Abbreviations

Executive Summary

The evaluation of the CA in Support of the Remediation of the former MCL radar sites in Ontario was conducted by Chief Review Services (CRS) between September and December 2013. The Evaluation is a component of the Department of National Defence (DND) Five-Year Evaluation Plan (2012/13 to 2016/17). It was conducted according to the Treasury Board (TB) 2009 Policy on Evaluation and the 2008 Policy on Transfer Payments.

The CA assists the Province of Ontario with remediating 11 former MCL radar sites in Ontario. The objective is to restore the radar sites to a condition that reduces the potential for future risks to human health, public safety and the environment. This remediation includes:

• removing the non-hazardous materials and their treatment on- or off-site depending on cost; and
• removing, shipping and disposing of the hazardous materials and contaminants to a regulated off-site disposal location.

In 2009, the federal government agreed to contribute $30 million over five years towards the estimated total of $60 million in expenditures to complete the remediation work. As of March 2014, two of the 11 sites have received Site Closure/Completion reports and have been accepted by the stakeholders. Three of the sites are in progress and considered nearly done. The remaining six sites are about to be remediated. All remediation projects are expected to be completed by the end of FY 2014/15, with inspections performed shortly thereafter. Accordingly, funds will remain to be spent in FY 2015/16 to complete the Site Closure/Completion reports and arrange for stakeholder acceptance. Therefore, the Program will not meet its objective of spending the Contribution monies by the end of FY 2014/15.

Of note, the CA fostered a high level of trust between disparate groups, which has benefitted all stakeholders.

Key Findings and Recommendations

Key Finding 1: The CA was successful in helping to remediate the contaminated radar sites in Ontario through the federal government’s past actions.

Key Finding 2: The CA requires an amendment to extend the Completion date beyond FY 2014/15 to complete the remediation of the remaining contaminated sites.

Key Finding 3: The use of an experienced project advisor with expertise in remediating contaminated sites was a key factor behind the project’s success.

Recommendation 1: The Assistant Deputy Minister (Infrastructure and Environment) (ADM(IE)) should ensure that the Ministry of Natural Resources (MNR) prepares the completion attestations for all 11 sites by the end of summer 2014, and that the attestations are received by March 31, 2015.

Recommendation 2: As the CA in its current form expires at the end of FY 2014/15, and as the MNR must complete the closure of the remaining seven sites in the summer of 2014 and subsequently prepare the appropriate site closure documents, the evaluation team recommends extending the CA to March 31, 2016.

Note: Refer to Annex A—Management Action Plan for the recommendations and management responses.

1.0 Introduction

1.1 Profile of the Mid-Canada Line Contribution Agreement

1.2.1 Resources

The total expenditures covered by the Evaluation are approximately $33 million. This figure represents the sum of the CA ($30 million), plus the cost of project support from Defence Construction Canada (DCC) ($2 million), and the cost of DND administrative and project management staff and their overhead for FYs 2009/10 to 2013/14 (approximately $1 million).

The CA spending over the period covered by the Evaluation (FYs 2007/08 to 2013/14) was $24.58 million. DND operations and maintenance (O&M) spending for DCC to support program management over the same period was $1.875 million. Expected disbursements of $4.95 million are foreseen for FY 2014/15. The DND estimates that the O&M spending will be $150,000 in FY 2014/15.

1.2.2 Issues and Questions

According to the TB Directive of the Evaluation Function, the core issues and questions addressed by the evaluation team are the Program’s relevance and performance (i.e., effectiveness, efficiency and economy). This includes:

1. To what extent does the CA address a demonstrable need?
2. Does the CA align with federal government roles and responsibilities?
3. Does the CA align with federal government priorities and the DND/Canadian Armed Forces (CAF) strategic outcomes?
4. To what extent does the CA meet expected outcomes?
5. Does the Program represent value for money?

See Annex D—Evaluation Matrix for the complete evaluation matrix.

2.0 Findings and Recommendations

The evaluation findings and recommendations are outlined in Sections 2.1 through 2.5.

2.1 Continued Need

The evaluation team determined that the principal need for this Program will remain until all 11 sites have been environmentally remediated (FY 2015/16), to protect human health and the environment.

Independent reviews³ of the 11 Class 1 and Class 2 MCL radar sites in Ontario determined that contaminants, including polychlorinated biphenyls, total petroleum hydrocarbons, asbestos, and heavy metals, had discharged into the environment. The sources of the contamination were buildings, paints, electrical equipment, diesel and oil spills, and pesticide spraying. The contaminants were deemed to have had an existing or potential adverse impact on human health and the environment.

2.2 Alignment with Federal Roles and Responsibilities

The CA aligns with federal roles and responsibilities. In April 2005, the federal government, through the Shared Responsibility and Contaminated Sites Framework, accepted responsibility for the long-term management of federal and shared responsibility contaminated sites. Under this Framework, the Federal Contaminated Site Assistant Deputy Minister Steering Committee agreed that the federal government would negotiate a cost-sharing agreement with the Province of Ontario to clean up former federally-owned or operated sites.

Further, it is appropriate for DND to provide the federal component of the funding. In response to several iterations of environmental sustainability, culminating in the implementation of the Federal Sustainable Development Strategy (FSDS) over the period 2010–2013,⁴ DND/CAF developed the Defence Environmental Strategy (DES) 2013. The vision of the DES is to manage Defence establishments, and training and operating areas, so that any effects on the environment are minimized—while ensuring military readiness. The DES provides direction to DND/CAF on how to accomplish this, and aligns with DND’s/CAF’s commitment to remediate sites that pose the highest risk to human health or the environment.

2.3 Alignment with Government Priorities

The CA also aligns with current Government of Canada priorities. Remediating land contaminated by past actions of the federal government has been a federal government priority since the early 1980s. In 2005, the Shared Responsibility of Contaminated Sites Framework was established. This Framework set out criteria under which the federal government would consider funding a portion of the clean-up costs of contaminated sites on non-federal land, to the extent that the contamination was a direct result of federal government activities or operations.

Aboriginal consultation and environmental stewardship also continue to be priorities for the federal government. The 2013 Speech from the Throne announced its continued commitment to work in partnership with Aboriginal peoples to create healthy, prosperous and self-sufficient communities. The government’s 2013–2016 FSDS, released November 2013, emphasizes the importance of the federal government, the provinces, the territories and Aboriginal groups working together to protect Canadian landscapes, seascapes, ecosystems and species at risk.

2.4 Achievement of Expected Outcomes (Effectiveness)

Key Finding 1: The CA was successful in helping to remediate the contaminated radar sites in Ontario through the federal government’s past actions.

The effectiveness of the CA was assessed by applying appropriate performance measures and key performance indicators against expected outcomes. These outcomes included:

• the quality (i.e., effectiveness) of site remediation;
• stakeholder acceptance (i.e., satisfaction) with the remediation; and
• compliance with the Program’s terms and conditions.

These criteria are shown as the immediate and intermediate outcomes on the Program’s logic model (see Annex C).

Data for the performance measures was obtained from reports, documents, studies, financial reports, documents provided by Directorate of Environmental Engineering Management staff, and several interviews with the program manager and the primary recipient stakeholder. Based upon an analysis of this information, the overall effectiveness of the CA was determined.

2.4.1 Quality Remediation

To assess the effectiveness of the remediation work, the evaluation team applied performance measures as shown in Table 1.

Table 1. Effective Remediation Performance Measures

Based upon the data from these performance measures, the evaluation team determined that:

• the Program has demonstrated sound remediation practices;
• the Program is complying with regulations and, for the most part, demonstrating sound project management practices; and
• all work and contractual obligations are being met.

2.4.1.1 Compliance with Environmental Assessments

All 11 sites were required to have an EA performed. Based upon the EAs, the various permits for the conduct of work were then applied for, and the Ontario Ministry of the Environment (OMOE) monitored the work to ensure that compliance was conducted. As permit responsibilities rest with the MNR at the project level, DND did not monitor the status of individual permits. The DND observed that the OMOE representatives were routinely present at Project and Program meetings. The MNR stated that all required permits were issued to contractors post-award.

Indicators for compliance checks demonstrated that

• an EA process was completed and registered for 11 sites;
• signed permits were issued by regulators and obtained for the 11 sites;
• Site Completion reports were completed within 90 days of the site being completed, and were submitted to and reviewed by the DGE or Director Environmental Protection (D Env P); and
• confirmatory sampling occurred to document compliance with standards, guidelines, recordkeeping and information.

2.4.1.2 Use of Integrated Planning and Business Cases

The completion of management plans by MNR and their review by DND and DCC ensured that work planned for was accomplished in a timely manner. The evaluation team found that there was an effective use of project management plans, integrated project planning, and business cases.

The MNR committed to and prepared working copies of eight overarching management plans. The site-specific plans were updated by the MNR annually and reviewed by DND for advice periodically as requested by the MNR. Requirements stemming from the management plans (e.g., health and safety, and emergency procedures) were accounted for in the MNR’s procurement of site clean-up services. The MNR occasionally used consultant support to update the project management plans.

Amendment number 2 to Appendix E of the CA changed the provision to two or three Milestone reports/invoices per year. The evaluation team also found that, in general terms, those due dates were met (i.e., with some flexibility depending on when the contractor’s final deliverables were completed and submitted to the Ontario MNR).

2.4.1.3 Quality Assurance Performed

For each site clean-up, the MNR engaged specialist quality management consultants to work with the remediation/camp contractors. These consultants ensured quality assurance/control and compliance with Health and Safety plans and the clean-up protocols related to worker safety and contaminated soils and materials. This included monitoring, sampling and laboratory analysis. The remediation and quality assurance/control specialist services provided reports with methodology, including quality control methodology and test results, which form appendices to the Site Completion Report.

2.4.1.4 Application of Lessons Learned

Lessons learned involves collecting, assessing, validating and using data to determine the failure points and deliverables of a project. +There should be a method or process for integrating those lessons into the organization. The success of a lessons learned system depends on buy-in from the stakeholders.

The evaluation team concluded that there was an application of lessons learned through implementing a Quality Assurance (QA) action list and developing business plans. A QA action list and the business plans were developed at the project and program levels. A Responsibility Assignment Matrix was developed, minutes were taken at project QA and Risk Management meetings, and a Project Management Plan was developed.

2.4.2 Stakeholder Acceptance

A key component of the CA was to obtain acceptance by all stakeholders of each remediated site. This includes, principally, FN bands, Ontario Parks, local municipalities, and other federal government departments designated as responsible authorities through the EA process.

To assess stakeholder acceptance, the evaluation team conducted interviews and document reviews to gauge the overall acceptance of the remediation.

Key indicators included:

• stakeholder verification of the remediation process and results; and
• stakeholder satisfaction.

The measures and their indicators/data sources are shown on the Evaluation Matrix. See Annex D—Evaluation Matrix.

2.4.2.1 Stakeholder Verification of the Remediation Process and Results

The evaluation team found, through various documentation reviews, confirmation that verification instruments were in place for all remediation sites. The documents included Milestone reports, Site Completion reports, Project Summary reports, Site Visit Inspection reports, and the minutes for status meetings. Evaluation interviews also confirmed that the stakeholders participated in the Program and accepted the results.

2.4.2.2 Stakeholder Satisfaction

Effective communications assist in understanding, build trust and respect, and create an environment where creative ideas and problem-solving can flourish. Effective communications can reduce conflict and frustration. The evaluation team found that effective use of engagement was met through communication and acceptance of all stakeholders.

First Nations

The MNR made satisfactory efforts to obtain acceptance from the FN stakeholders regarding site remediation activities. The MNR sought FN traditional knowledge as part of their site investigation research. The EA process included consultation with FNs through public meetings and was reflected in the meeting minutes.

A significant component of the Contribution Agreement was to provide the FN people with opportunities and benefits. Aboriginal people from the local communities formed a large part of the work force. Their direct involvement in the remediation helped to communicate the scope and quality of the project. Further, the MNR held tours for FN elders at the start and end of the site clean-ups to ensure that FNs input was considered. The MNR also sought FN input at the semi-annual program meetings and routine teleconferences. Lastly the MNR stated that FN religious closure ceremonies were carried out at some of the sites.

Ontario Parks

The MNR obtained Ontario Parks acceptance regarding site remediation efforts. The MNR communicated standards and negotiated for engineered on-site landfills with Ontario Parks. During landfill construction, MNR site representatives included an Ontario Parks member. The MNR followed the provincial EA process and obtained the necessary permits.

Department of National Defence and Defence Construction Canada

Communication between the MNR and DND/DCC was open and transparent. Acceptance by the DND of the MNR’s site remediation efforts was facilitated by the MNR’s invoice/milestone reports, site visits, regular meetings, review of plans/reports and sharing of advice. The MNR stated that, in large part, the success of the project was due to DCC’s efforts in project management and knowledge transfer.

2.4.3 Compliance with the Contribution Agreement

The evaluation team considered the following factors in assessing the project’s compliance with the CA’s terms and conditions:

• adherence to the schedule and contract; and
• attestation of project completion.

The measures and their indicators or data sources are shown on the Evaluation Matrix (See Annex D).

2.4.3.1 Adherence to Schedule and Contract

Reasonable progress is underway on all sites. Most of the work is completed. Two sites are closed, and the remaining nine sites are on track for completion by the end of FY 2014/15. If the MNR continues with the best practices that it has used, it is expected that these remaining nine sites will be accepted by the stakeholders by the end of FY 2014/15.

Remediation activities funded by the CA were completed at eight radar sites by the fall of FY 2013/14. Three sites remain to be remediated in FY 2014/15.

Table 2 provides the MCL radar site completion status as of 10 December 2013.

Table 2. Site Remediation Status

2.4.3.2 Attestation of Project Completion

The evaluation team found that the three completed sites all met site completion attestation. Confirmatory sampling confirmed that the environmental component of the site remediation was completed according to the applicable criteria. It is expected that the remaining sites will be accepted by the stakeholders and will comply with the terms and conditions of the CA by the end of FY 2014/15.

Key Finding 2: The CA requires an amendment to extend the Completion date beyond FY 2014/15 to complete the remediation of the remaining contaminated sites.

2.4.4 Remediated Sites

The ultimate outcome of the CA is that the 11 sites be remediated to the point where they would be deemed acceptable for safe use by all stakeholders. While the original schedule was not met, the delays experienced were neither unforeseen nor unreasonable given the uncertainty of the extent of contamination.

2.5 Demonstration of Efficiency and Economy

The evaluation team considered the following criteria to assess the overall efficiency and economy of the Program:

• value for money of remediation work; and
• cost of overhead and program management.

The information was gathered by reviewing program documents and reports and through key informant interviews.

2.5.1 Value for Money of Remediation Work

To assess these criteria, the following indicators were considered:

• use of competitive contracting;
• adherence to cost estimates;
• expertise of project management; and
• cost of overhead and program management.

2.5.1.1 Use of competitive contracting

Through examination of contractual documents and the interview process, the evaluation team noted that sound procurement practices were utilized throughout the project. This included using competitive contracting to ensure market prices, bundling work for all sites to minimize costs per site and establishing fixed unit prices for extra work.

The evaluation team was not able to conduct an exhaustive comparative analysis with other remediation projects (e.g., the DEW Line Clean-up, Goose Bay or TCE [or trichloroethylene] Valcartier) that were carried out by DND, due to the unique scope and complexity of each remediation project.

However, although direct comparisons are difficult, even with the cost overrun the average cost per remediation of the MCL sites appears to be 40 percent of the cost of similar remediation work performed on earlier projects, such as those of the DEW line.

Table 3. Cost Comparison

2.5.1.2 Adherence to Cost Estimates

The CA transfers $30 million from the federal government to the Province of Ontario over the period FYs 2008/09 to 2014/15. In addition, DND planned to use $2.25 million of the O&M money for program/project management and administration. There was no accounting for salary, office space and overhead for the program/project management and oversight of staff within ADM(IE).

Table 4. Vote 1 and Vote 10 Monies

CA spending over the period covered by the evaluation team (FYs 2007/08 to 2013/14) was $24.58 million⁷ leaving $5.42 million unspent as at the end of FY 2013/14.

Expected disbursements from the CA of $4.95 million are foreseen for FY 2014/15.

The evaluation team found that the project budget was not adhered to. The initial estimate of $60 million for the total remediation of 11 sites was developed in 1999 by SNC-Lavalin. In 2003, the MNR revised the initial scope of work and with SNC-Lavalin came up with a new estimate, in 2007, of $85 million.

Meanwhile, discussions were underway between the MNR and DND as to what constituted an eligible cost and what was not. The final agreement in 2008 was that, of the estimated $85 million, $60 million was eligible for the DND CA.

In 2013, the MNR indicated that their estimate of the final cost was going to be between $91 million and $96 million, which is approximately 7 to 12 percent higher than the 2007 estimate. Given the uncertain nature of this work and the duration it took to complete, the actual estimation process appears to be fairly accurate and reasonable.

2.5.1.3 Expertise of Project Management

The MNR staff assigned to the project had no previous experience in managing, nor in administering or conducting a contaminated site remediation project. All of their previous experience was as a Crown land custodian. Consequently, the Project Advisor role was contracted to a third party who had extensive experience in contaminated site remediation in remote areas. In hindsight, this was a wise decision. Acting as teacher and mentor to the MNR staff, it was the Project Advisor’s experience and expertise in contaminated site remediation that allowed the project to succeed.

Key Finding 3: The use of an experienced project advisor with expertise in remediating contaminated sites in remote areas was a key factor behind the project’s success.

2.5.1.4 Cost of Overhead and Program Management

The evaluation team determined through financial review and analysis that the DND overhead cost was $1.875 million on $24.56 million in expenditures. This represents a 7.6 percent administrative overhead charge, which is also not unreasonable for a program of this nature.

2.6 Conclusion

As established in the Relevance section, the CA conformed to the federal government’s commitment to fund the remediation of shared-responsibility contaminated sites. This assessment is based on the evidence provided by completed environmental research and the federal government’s policy decisions.

As established in the Performance section, the CA achieved the expected outcomes during the evaluation period. Two of the eleven sites are complete and have been accepted by the stakeholders, two sites are complete but require stakeholder acceptance, and seven remaining sites are considered partially done. The project is expected to successfully remediate all eleven required sites by end of FY 2014/15.

Of note, the Contribution Agreement fostered a high level of trust between disparate groups, which has benefitted all stakeholders.

Annex A—Management Action Plan

Management Action

ADM(IE) will collaborate with the MNR to revise the processes that will be included in the Amendment to the CA, to ensure attestations are completed for the 11 sites and received by 31 March 2015.

OPI: ADM(IE) / DGPR

Target Date: December 2014 for process review

Management Action

ADM(IE) will seek from the MNR an extension to March 31, 2016, of the current CA. The expected timeline to produce the Amendment to the CA is December 2014.

OPI: ADM(IE) / DGPR

Target Date: December 2014 for Amendment to the CA

Annex B—Evaluation Methodology and Limitations

1.0 Methodology

The evaluation team used multiple lines of evidence and complementary qualitative and quantitative research methods to help ensure the reliability of information and data to support the evaluation findings. The methodology established a consistent approach in collecting and analyzing the data to support the evaluation findings, conclusions and recommendations. Based on the evidence obtained from available sources, the evaluation team reviewed the achievement of the expected outcomes, and the Program’s efficiency and economy, to develop a balanced picture of the relevance and performance of the CA. Information and data were correlated to each evaluation question and corresponding indicators. To ensure the validity of the data captured, the evaluation team used a data triangulation approach. The evaluation team also streamlined the key components of the original G&C logic model, and removed the outcomes considered as indicators. (See the Logic Model in Annex C).

1.1 Overview of Data Collection Methods

The data collection methods were selected based on the data required to address the performance indicators. The evaluation team used the following data collection methods to gather qualitative and quantitative data for each type of operation:

• literature and document review;
• key informant interviews; and
• administrative and financial data reviews.

1.1.1 Documents and Files Review

A preliminary document review was conducted as part of the Planning phase by the evaluation team to garner a foundational understanding of the CA. A comprehensive document review was undertaken as part of the conduct phase by the evaluation team, focusing on the relevance and the performance of the CA activities.

The following documents were reviewed by the evaluation team during the conduct phase:

• program documents:
  • Treasury Board submission, the CA annual reports and other activity reports;
• administrative documents:
  • CA financial reports, correspondence exchange; and
• accountability documents:
  • DND/CAF Reports on Plans and Priorities, DND/CAF Departmental Performance Reports, Canada First Defence Strategy and Speech from the Throne.

The document review was conducted using a customized template organized according to the evaluation questions and indicators.

1.1.2 Key Informant Interviews

Key informant interviews and information sessions scheduled with CA stakeholders, who were directly or indirectly involved in delivering the Program, served as an important source of qualitative information.

Individual interviews were conducted in person with the DND project managers, the DCC project advisor, and the ADM(IE) Comptroller. Telephone interviews were conducted with the MNR staff.

1.1.3 Review of Financial and Administrative Data

The CA financial data was reviewed to determine the degree of efficiency and economy of the Program’s activities and outputs. The data, which covered five years from 2009 to 2013, was extracted from the CA financial reports.

2.0 Limitations

The following table shows the limitations related to the sources and the mitigation strategies applied to them.

Table B-1. Evaluation Limitations and Mitigation Strategies

Annex C—Logic Model

Annex D—Evaluation Matrix

Table D-1. Evaluation Matrix

___________________________________________________________________________________________________________________________

Footnote 1 The other two lines were the DEW line along the 69th parallel, and the Pinetree line along the 53rd parallel in the West and the 50th parallel in the East. Unlike the United States-Canada jointly-operated Pinetree line and DEW line, the MCL was funded and operated entirely by the Royal Canadian Air Force.

Footnote 2 As established by the Canadian Council Ministers of the Environment in 1992.

Footnote 3 There were at least 5 environmental reviews and assessments of the 11 sites over the period 1980–2000.

Footnote 4 Canada’s Green Plan (1990); DND Sustainable Development Strategy Iterations 1–4 (1997–2003); Federal Contaminated Sites Action Plan (2004); CRS Audit of Sustainable Development Strategy (2009); Director General Environment (DGE) Program Review (2010); FSDS Implementation (2010–2013); and DES Development (2011–2013).

Footnote 5 Based on a $30 million Contribution. If an estimated total cost of $110 million is used, the cost per site is $10 million—of which the federal government’s share is $2.7 million for each site.

Footnote 6 Estimated spending for FY 2014/15.

Footnote 7 The original schedule called for spending $29,750,000 by end of FY 2013/14.