Evaluation of the Contribution Agreement in support of the Remediation of the Former Mid-Canada Line Radar Sites in Ontario
Alternate Formats
Assistant Deputy Minister (Review Services)
June 2014
1258-208 (CRS)
Reviewed by CRS in accordance with the Access to Information Act (AIA). Information UNCLASSIFIED.
Table of Contents
-
Acronyms and Abbreviations
-
Executive Summary
-
1.0 Introduction
-
2.0 Findings and Recommendations
-
Annex A—Management Action Plan
-
Annex B—Evaluation Methodology and Limitations
-
Annex C—Logic Model
-
Annex D—Evaluation Matrix
- CAF
Canadian Armed Forces
- CA
Contribution Agreement
- D Env P
Director Environmental Protection
- DCC
Defence Construction Canada
- DES
Defence Environmental Strategy
- DEW
Distant Early Warning
- DGE
Director General Environment
- DND
Department of National Defence
- EA
Environmental Assessment
- ADM(IE)
Assistant Deputy Minister (Infrastructure and Environment)
- FY
-
Fiscal Year
- FN
-
First Nation
- FSDS
-
Federal Sustainable Development Strategy
- MCL
-
Mid-Canada Line
- MNR
-
Ministry of Natural Resources
- OMOE
-
Ontario Ministry of the Environment
- O&M
-
Operations and Maintenance
- QA
-
Quality Assurance
- TB
-
Treasury Board
Executive Summary
Overall Assessment
- The Contribution Agreement (CA) in Support of the Remediation of the former Mid-Canada Line (MCL) radar sites in Ontario should meet all of its deliverables by Fiscal Year (FY) 2015/16.
- While the project did not meet the original anticipated schedule, the overall progress was not unreasonable given the nature of this work.
- The management practices used on this Program have minimized cost overruns and kept the project expenditures to within a reasonable range of the original estimates.
The evaluation of the CA in Support of the Remediation of the former MCL radar sites in Ontario was conducted by Chief Review Services (CRS) between September and December 2013. The Evaluation is a component of the Department of National Defence (DND) Five-Year Evaluation Plan (2012/13 to 2016/17). It was conducted according to the Treasury Board (TB) 2009 Policy on Evaluation and the 2008 Policy on Transfer Payments.
The CA assists the Province of Ontario with remediating 11 former MCL radar sites in Ontario. The objective is to restore the radar sites to a condition that reduces the potential for future risks to human health, public safety and the environment. This remediation includes:
- removing the non-hazardous materials and their treatment on- or off-site depending on cost; and
- removing, shipping and disposing of the hazardous materials and contaminants to a regulated off-site disposal location.
In 2009, the federal government agreed to contribute $30 million over five years towards the estimated total of $60 million in expenditures to complete the remediation work. As of March 2014, two of the 11 sites have received Site Closure/Completion reports and have been accepted by the stakeholders. Three of the sites are in progress and considered nearly done. The remaining six sites are about to be remediated. All remediation projects are expected to be completed by the end of FY 2014/15, with inspections performed shortly thereafter. Accordingly, funds will remain to be spent in FY 2015/16 to complete the Site Closure/Completion reports and arrange for stakeholder acceptance. Therefore, the Program will not meet its objective of spending the Contribution monies by the end of FY 2014/15.
Of note, the CA fostered a high level of trust between disparate groups, which has benefitted all stakeholders.
Key Findings and Recommendations
Key Finding 1: The CA was successful in helping to remediate the contaminated radar sites in Ontario through the federal government’s past actions.
Key Finding 2: The CA requires an amendment to extend the Completion date beyond FY 2014/15 to complete the remediation of the remaining contaminated sites.
Key Finding 3: The use of an experienced project advisor with expertise in remediating contaminated sites was a key factor behind the project’s success.
Recommendation 1: The Assistant Deputy Minister (Infrastructure and Environment) (ADM(IE)) should ensure that the Ministry of Natural Resources (MNR) prepares the completion attestations for all 11 sites by the end of summer 2014, and that the attestations are received by March 31, 2015.
Recommendation 2: As the CA in its current form expires at the end of FY 2014/15, and as the MNR must complete the closure of the remaining seven sites in the summer of 2014 and subsequently prepare the appropriate site closure documents, the evaluation team recommends extending the CA to March 31, 2016.
Note: Refer to Annex A—Management Action Plan for the recommendations and management responses.
1.1.1 Background
The MCL CA is a $30 million contribution from DND to the Province of Ontario to remediate the 11 contaminated sites in the Ontario portion of the MCL radar systems.
The MCL was one of three major radar chains constructed by the United States and Canadian governments in the 1950s.1 The purpose of these radar chains was to provide Canada and the United States with a radar surveillance network to augment the Canadian air defence system.
In 1957, the MCL consisted of 98 radar stations sited along the 55th parallel from Dawson Creek, British Columbia, to Hopedale, Labrador. Seventeen of the MCL radar sites were in Ontario.
When the Distant Early Warning (DEW) line became operational in 1957, and the Soviet Union moved its offensive capability to Inter-Continental Ballistic Missiles, it became clear that the MCL and Pinetree radar systems were of limited use. The decommissioning of the Ontario sites began in 1964, and the last site closed in 1965. The Crown Assets Disposal Corporation transferred the buildings and land to the Province of Ontario between 1966 and 1969.
Under the National Classification System2 for remediating the contaminated sites, 12 of the 17 sites in Ontario were deemed to require remediation. One of the 12 sites, Fort Albany, was remediated by the Province of Ontario in 2002.
In 2005, the total cost of remediating the 11 remaining Ontario MCL radar sites was estimated at $60 million. In March 2007, DND negotiated a CA with the Ontario MNR to provide financial support of up to 50 percent of the estimated costs of the remediation. The DND’s funding of $30 million was approved in 2009.
Figure 1. Location of the Early Warning Radar Lines. Map of Canada showing the location of the three lines of radar sites built during the 1950s.
Source: Wikipedia
1.1.2 Program Description and Objectives
The objective of the MCL CA is to assist the Province of Ontario with remediating the former MCL radar sites, which includes:
- removing and treating the non-hazardous materials on- or off-site depending on cost; and
- removing, shipping and disposing the hazardous materials and contaminants to a regulated off-site disposal location.
1.1.3 Stakeholders and Beneficiaries
The stakeholders that are involved include the following:
- the Ontario MNR, which is responsible for all necessary permits, Certificates of Approval, conducting the necessary environmental assessments (EAs), and for carrying out all activities deemed necessary by federal and provincial regulations to remediate the sites;
- DND, which is responsible for carrying out the federal responsibilities associated with the CA; and
- as specified in the CA, local communities affected by the contaminated sites—First Nations (FN) bands, the public, and local municipalities—along with other federal government departments designated as responsible authorities through the EA process.
1.2 Evaluation Scope
The evaluation team considered the issues of relevance and performance (i.e., effectiveness and efficiency) based on the terms and conditions of the $30 million MCL CA, and according to DND’s Program Alignment Architecture.
The 2013 Program Alignment Architecture places the MCL CA in sub-sub-activity level 4.3.5Real Property—Environment and Remediation. This last links to the Strategic Outcome—Defence Remains Continually Prepared to Deliver National Defence and Defence Services in Alignment with Canadian Interests and Values.
Figure 2. Map of the MCL Radar Sites in Ontario. Map showing the location of the MCL radar sites along the south shore of Hudson Bay.
Source: ADM(IE)
1.2.1 Resources
The total expenditures covered by the Evaluation are approximately $33 million. This figure represents the sum of the CA ($30 million), plus the cost of project support from Defence Construction Canada (DCC) ($2 million), and the cost of DND administrative and project management staff and their overhead for FYs 2009/10 to 2013/14 (approximately $1 million).
The CA spending over the period covered by the Evaluation (FYs 2007/08 to 2013/14) was $24.58 million. DND operations and maintenance (O&M) spending for DCC to support program management over the same period was $1.875 million. Expected disbursements of $4.95 million are foreseen for FY 2014/15. The DND estimates that the O&M spending will be $150,000 in FY 2014/15.
1.2.2 Issues and Questions
According to the TB Directive of the Evaluation Function, the core issues and questions addressed by the evaluation team are the Program’s relevance and performance (i.e., effectiveness, efficiency and economy). This includes:
- To what extent does the CA address a demonstrable need?
- Does the CA align with federal government roles and responsibilities?
- Does the CA align with federal government priorities and the DND/Canadian Armed Forces (CAF) strategic outcomes?
- To what extent does the CA meet expected outcomes?
- Does the Program represent value for money?
See Annex D—Evaluation Matrix for the complete evaluation matrix.
2.0 Findings and Recommendations
The evaluation findings and recommendations are outlined in Sections 2.1 through 2.5.
2.1 Continued Need
The evaluation team determined that the principal need for this Program will remain until all 11 sites have been environmentally remediated (FY 2015/16), to protect human health and the environment.
Independent reviews3 of the 11 Class 1 and Class 2 MCL radar sites in Ontario determined that contaminants, including polychlorinated biphenyls, total petroleum hydrocarbons, asbestos, and heavy metals, had discharged into the environment. The sources of the contamination were buildings, paints, electrical equipment, diesel and oil spills, and pesticide spraying. The contaminants were deemed to have had an existing or potential adverse impact on human health and the environment.
2.2 Alignment with Federal Roles and Responsibilities
The CA aligns with federal roles and responsibilities. In April 2005, the federal government, through the Shared Responsibility and Contaminated Sites Framework, accepted responsibility for the long-term management of federal and shared responsibility contaminated sites. Under this Framework, the Federal Contaminated Site Assistant Deputy Minister Steering Committee agreed that the federal government would negotiate a cost-sharing agreement with the Province of Ontario to clean up former federally-owned or operated sites.
Further, it is appropriate for DND to provide the federal component of the funding. In response to several iterations of environmental sustainability, culminating in the implementation of the Federal Sustainable Development Strategy (FSDS) over the period 2010–2013,4 DND/CAF developed the Defence Environmental Strategy (DES) 2013. The vision of the DES is to manage Defence establishments, and training and operating areas, so that any effects on the environment are minimized—while ensuring military readiness. The DES provides direction to DND/CAF on how to accomplish this, and aligns with DND’s/CAF’s commitment to remediate sites that pose the highest risk to human health or the environment.
2.3 Alignment with Government Priorities
The CA also aligns with current Government of Canada priorities. Remediating land contaminated by past actions of the federal government has been a federal government priority since the early 1980s. In 2005, the Shared Responsibility of Contaminated Sites Framework was established. This Framework set out criteria under which the federal government would consider funding a portion of the clean-up costs of contaminated sites on non-federal land, to the extent that the contamination was a direct result of federal government activities or operations.
Aboriginal consultation and environmental stewardship also continue to be priorities for the federal government. The 2013 Speech from the Throne announced its continued commitment to work in partnership with Aboriginal peoples to create healthy, prosperous and self-sufficient communities. The government’s 2013–2016 FSDS, released November 2013, emphasizes the importance of the federal government, the provinces, the territories and Aboriginal groups working together to protect Canadian landscapes, seascapes, ecosystems and species at risk.
2.4 Achievement of Expected Outcomes (Effectiveness)
Key Finding 1: The CA was successful in helping to remediate the contaminated radar sites in Ontario through the federal government’s past actions.
The effectiveness of the CA was assessed by applying appropriate performance measures and key performance indicators against expected outcomes. These outcomes included:
- the quality (i.e., effectiveness) of site remediation;
- stakeholder acceptance (i.e., satisfaction) with the remediation; and
- compliance with the Program’s terms and conditions.
These criteria are shown as the immediate and intermediate outcomes on the Program’s logic model (see Annex C).
Data for the performance measures was obtained from reports, documents, studies, financial reports, documents provided by Directorate of Environmental Engineering Management staff, and several interviews with the program manager and the primary recipient stakeholder. Based upon an analysis of this information, the overall effectiveness of the CA was determined.
2.4.1 Quality Remediation
To assess the effectiveness of the remediation work, the evaluation team applied performance measures as shown in Table 1.
Table 1. Effective Remediation Performance Measures
Table Summary:
This table shows the performance measures, indicators and targets, by percentage, of site remediations.
Outcome | Performance Measures | Indicators | Targets (percentage) |
---|---|---|---|
Site remediation | Compliance with EAs | Percentage of sites with EAs performed and permits obtained | 100 |
Use of integrated project planning and business cases | Percentage of sites with plans and business cases | 100 | |
QA performed | Percentage of sites with a QA process | 100 | |
Application of lessons learned | Demonstrated use |
Based upon the data from these performance measures, the evaluation team determined that:
- the Program has demonstrated sound remediation practices;
- the Program is complying with regulations and, for the most part, demonstrating sound project management practices; and
- all work and contractual obligations are being met.
2.4.1.1 Compliance with Environmental Assessments
All 11 sites were required to have an EA performed. Based upon the EAs, the various permits for the conduct of work were then applied for, and the Ontario Ministry of the Environment (OMOE) monitored the work to ensure that compliance was conducted. As permit responsibilities rest with the MNR at the project level, DND did not monitor the status of individual permits. The DND observed that the OMOE representatives were routinely present at Project and Program meetings. The MNR stated that all required permits were issued to contractors post-award.
Indicators for compliance checks demonstrated that
- an EA process was completed and registered for 11 sites;
- signed permits were issued by regulators and obtained for the 11 sites;
- Site Completion reports were completed within 90 days of the site being completed, and were submitted to and reviewed by the DGE or Director Environmental Protection (D Env P); and
- confirmatory sampling occurred to document compliance with standards, guidelines, recordkeeping and information.
2.4.1.2 Use of Integrated Planning and Business Cases
The completion of management plans by MNR and their review by DND and DCC ensured that work planned for was accomplished in a timely manner. The evaluation team found that there was an effective use of project management plans, integrated project planning, and business cases.
The MNR committed to and prepared working copies of eight overarching management plans. The site-specific plans were updated by the MNR annually and reviewed by DND for advice periodically as requested by the MNR. Requirements stemming from the management plans (e.g., health and safety, and emergency procedures) were accounted for in the MNR’s procurement of site clean-up services. The MNR occasionally used consultant support to update the project management plans.
Amendment number 2 to Appendix E of the CA changed the provision to two or three Milestone reports/invoices per year. The evaluation team also found that, in general terms, those due dates were met (i.e., with some flexibility depending on when the contractor’s final deliverables were completed and submitted to the Ontario MNR).
2.4.1.3 Quality Assurance Performed
For each site clean-up, the MNR engaged specialist quality management consultants to work with the remediation/camp contractors. These consultants ensured quality assurance/control and compliance with Health and Safety plans and the clean-up protocols related to worker safety and contaminated soils and materials. This included monitoring, sampling and laboratory analysis. The remediation and quality assurance/control specialist services provided reports with methodology, including quality control methodology and test results, which form appendices to the Site Completion Report.
2.4.1.4 Application of Lessons Learned
Lessons learned involves collecting, assessing, validating and using data to determine the failure points and deliverables of a project. +There should be a method or process for integrating those lessons into the organization. The success of a lessons learned system depends on buy-in from the stakeholders.
The evaluation team concluded that there was an application of lessons learned through implementing a Quality Assurance (QA) action list and developing business plans. A QA action list and the business plans were developed at the project and program levels. A Responsibility Assignment Matrix was developed, minutes were taken at project QA and Risk Management meetings, and a Project Management Plan was developed.
2.4.2 Stakeholder Acceptance
A key component of the CA was to obtain acceptance by all stakeholders of each remediated site. This includes, principally, FN bands, Ontario Parks, local municipalities, and other federal government departments designated as responsible authorities through the EA process.
To assess stakeholder acceptance, the evaluation team conducted interviews and document reviews to gauge the overall acceptance of the remediation.
Key indicators included:
- stakeholder verification of the remediation process and results; and
- stakeholder satisfaction.
The measures and their indicators/data sources are shown on the Evaluation Matrix. See Annex D—Evaluation Matrix.
2.4.2.1 Stakeholder Verification of the Remediation Process and Results
The evaluation team found, through various documentation reviews, confirmation that verification instruments were in place for all remediation sites. The documents included Milestone reports, Site Completion reports, Project Summary reports, Site Visit Inspection reports, and the minutes for status meetings. Evaluation interviews also confirmed that the stakeholders participated in the Program and accepted the results.
2.4.2.2 Stakeholder Satisfaction
Effective communications assist in understanding, build trust and respect, and create an environment where creative ideas and problem-solving can flourish. Effective communications can reduce conflict and frustration. The evaluation team found that effective use of engagement was met through communication and acceptance of all stakeholders.
First Nations
The MNR made satisfactory efforts to obtain acceptance from the FN stakeholders regarding site remediation activities. The MNR sought FN traditional knowledge as part of their site investigation research. The EA process included consultation with FNs through public meetings and was reflected in the meeting minutes.
A significant component of the Contribution Agreement was to provide the FN people with opportunities and benefits. Aboriginal people from the local communities formed a large part of the work force. Their direct involvement in the remediation helped to communicate the scope and quality of the project. Further, the MNR held tours for FN elders at the start and end of the site clean-ups to ensure that FNs input was considered. The MNR also sought FN input at the semi-annual program meetings and routine teleconferences. Lastly the MNR stated that FN religious closure ceremonies were carried out at some of the sites.
Ontario Parks
The MNR obtained Ontario Parks acceptance regarding site remediation efforts. The MNR communicated standards and negotiated for engineered on-site landfills with Ontario Parks. During landfill construction, MNR site representatives included an Ontario Parks member. The MNR followed the provincial EA process and obtained the necessary permits.
Department of National Defence and Defence Construction Canada
Communication between the MNR and DND/DCC was open and transparent. Acceptance by the DND of the MNR’s site remediation efforts was facilitated by the MNR’s invoice/milestone reports, site visits, regular meetings, review of plans/reports and sharing of advice. The MNR stated that, in large part, the success of the project was due to DCC’s efforts in project management and knowledge transfer.
2.4.3 Compliance with the Contribution Agreement
The evaluation team considered the following factors in assessing the project’s compliance with the CA’s terms and conditions:
- adherence to the schedule and contract; and
- attestation of project completion.
The measures and their indicators or data sources are shown on the Evaluation Matrix (See Annex D).
2.4.3.1 Adherence to Schedule and Contract
Reasonable progress is underway on all sites. Most of the work is completed. Two sites are closed, and the remaining nine sites are on track for completion by the end of FY 2014/15. If the MNR continues with the best practices that it has used, it is expected that these remaining nine sites will be accepted by the stakeholders by the end of FY 2014/15.
Remediation activities funded by the CA were completed at eight radar sites by the fall of FY 2013/14. Three sites remain to be remediated in FY 2014/15.
Table 2 provides the MCL radar site completion status as of 10 December 2013.
Table 2. Site Remediation Status
Table Summary:
This table summarizes the remediation work status per site category.
Site | Site category | DND-funded Work | MNR-funded Work | Site Closure report | Attestation |
---|---|---|---|---|---|
60 | Relay site | Completed (October 2010) | Completed (October 2010) | Received (February 2011) | Received (April 2012) |
70 | Relay site | Completed (October 2010) | Completed (October 2010) | Received (March 2011) | Overdue (originally due 90 days within site completion) |
500 | Sector Control Station | Completed (September 2013) | Completed (September 2013) | In progress (due February 2014) | |
416 | Doppler Detection Station | Completed (October 2013) | Completed (October 2013) | ||
415 | Relay site | In progress | In progress | ||
418 | Doppler Detection Station | In progress | In progress | ||
421 | Doppler Detection Station | In progress | In progress | ||
503 | Doppler Detection Station | Completed (October 2013) | In abeyance, partially completed | ||
506 | Doppler Detection Station | Completed (October 2013) | In abeyance, partially completed | ||
424 | Doppler Detection Station | Completed (October 2013) | In abeyance, partially completed | ||
427 | Doppler Detection Station | Completed (October 2013) | In abeyance, partially completed |
2.4.3.2 Attestation of Project Completion
The evaluation team found that the three completed sites all met site completion attestation. Confirmatory sampling confirmed that the environmental component of the site remediation was completed according to the applicable criteria. It is expected that the remaining sites will be accepted by the stakeholders and will comply with the terms and conditions of the CA by the end of FY 2014/15.
Key Finding 2: The CA requires an amendment to extend the Completion date beyond FY 2014/15 to complete the remediation of the remaining contaminated sites.
2.4.4 Remediated Sites
The ultimate outcome of the CA is that the 11 sites be remediated to the point where they would be deemed acceptable for safe use by all stakeholders. While the original schedule was not met, the delays experienced were neither unforeseen nor unreasonable given the uncertainty of the extent of contamination.
2.5 Demonstration of Efficiency and Economy
The evaluation team considered the following criteria to assess the overall efficiency and economy of the Program:
- value for money of remediation work; and
- cost of overhead and program management.
The information was gathered by reviewing program documents and reports and through key informant interviews.
2.5.1 Value for Money of Remediation Work
To assess these criteria, the following indicators were considered:
- use of competitive contracting;
- adherence to cost estimates;
- expertise of project management; and
- cost of overhead and program management.
2.5.1.1 Use of competitive contracting
Through examination of contractual documents and the interview process, the evaluation team noted that sound procurement practices were utilized throughout the project. This included using competitive contracting to ensure market prices, bundling work for all sites to minimize costs per site and establishing fixed unit prices for extra work.
The evaluation team was not able to conduct an exhaustive comparative analysis with other remediation projects (e.g., the DEW Line Clean-up, Goose Bay or TCE [or trichloroethylene] Valcartier) that were carried out by DND, due to the unique scope and complexity of each remediation project.
However, although direct comparisons are difficult, even with the cost overrun the average cost per remediation of the MCL sites appears to be 40 percent of the cost of similar remediation work performed on earlier projects, such as those of the DEW line.
Table 3. Cost Comparison
Table Summary:
This table shows the total cost and cost per site of remediated sites per project.
Project | Number of Sites Remediated | Total Cost ($ million) | Cost per Site($ million) |
---|---|---|---|
DEW Line | 21 | 583.0 | 27.8 |
Goose Bay | 10 | 220.0 (estimate) | 22.0 |
Valcartier | 1 | 30.0 (estimate) | 30.0 |
MCL | 11 | 30.0 | 2.75 |
2.5.1.2 Adherence to Cost Estimates
The CA transfers $30 million from the federal government to the Province of Ontario over the period FYs 2008/09 to 2014/15. In addition, DND planned to use $2.25 million of the O&M money for program/project management and administration. There was no accounting for salary, office space and overhead for the program/project management and oversight of staff within ADM(IE).
Table 4. Vote 1 and Vote 10 Monies
Table Summary:
This table shows the budgeted and actual cost of O&M and Transfer Payments in thousands of dollars for FYs 2008/09 to 2014/15.
FY | Vote 1—O&M ($ 000s) | Vote 10—Transfer Payment ($ 000s) | ||||||
Budgeted Year | Budgeted Cumulative | Actual Year | Budgeted Cumulative | Budgeted Year | Budgeted Cumulative | Actual Year | Budgeted Cumulative | |
2008/09 | 0 | 0 | 400 | 400 | 317 | 317 | 0 | 0 |
2009/10 | 450 | 450 | 400 | 800 | 8,931 | 9,248 | 9,248 | 9,248 |
2010/11 | 450 | 900 | 400 | 1,200 | 6,595 | 15,843 | 3,457 | 12,705 |
2011/12 | 450 | 1,350 | 250 | 1,450 | 5,571 | 21,414 | 6,000 | 18,705 |
2012/13 | 450 | 1,800 | 225 | 1,675 | 6,315 | 27,729 | 3,575 | 22,280 |
2013/14 | 450 | 2,250 | 200 | 1,875 | 2,021 | 29,750 | 2,300 | 24,580 |
2014/15 | 0 | 2,250 | 1506 | 2,025 | 250 | 30,000 | 0 | 24,580 |
CA spending over the period covered by the evaluation team (FYs 2007/08 to 2013/14) was $24.58 million7 leaving $5.42 million unspent as at the end of FY 2013/14.
Expected disbursements from the CA of $4.95 million are foreseen for FY 2014/15.
The evaluation team found that the project budget was not adhered to. The initial estimate of $60 million for the total remediation of 11 sites was developed in 1999 by SNC-Lavalin. In 2003, the MNR revised the initial scope of work and with SNC-Lavalin came up with a new estimate, in 2007, of $85 million.
Meanwhile, discussions were underway between the MNR and DND as to what constituted an eligible cost and what was not. The final agreement in 2008 was that, of the estimated $85 million, $60 million was eligible for the DND CA.
In 2013, the MNR indicated that their estimate of the final cost was going to be between $91 million and $96 million, which is approximately 7 to 12 percent higher than the 2007 estimate. Given the uncertain nature of this work and the duration it took to complete, the actual estimation process appears to be fairly accurate and reasonable.
2.5.1.3 Expertise of Project Management
The MNR staff assigned to the project had no previous experience in managing, nor in administering or conducting a contaminated site remediation project. All of their previous experience was as a Crown land custodian. Consequently, the Project Advisor role was contracted to a third party who had extensive experience in contaminated site remediation in remote areas. In hindsight, this was a wise decision. Acting as teacher and mentor to the MNR staff, it was the Project Advisor’s experience and expertise in contaminated site remediation that allowed the project to succeed.
Key Finding 3: The use of an experienced project advisor with expertise in remediating contaminated sites in remote areas was a key factor behind the project’s success.
2.5.1.4 Cost of Overhead and Program Management
The evaluation team determined through financial review and analysis that the DND overhead cost was $1.875 million on $24.56 million in expenditures. This represents a 7.6 percent administrative overhead charge, which is also not unreasonable for a program of this nature.
2.6 Conclusion
As established in the Relevance section, the CA conformed to the federal government’s commitment to fund the remediation of shared-responsibility contaminated sites. This assessment is based on the evidence provided by completed environmental research and the federal government’s policy decisions.
As established in the Performance section, the CA achieved the expected outcomes during the evaluation period. Two of the eleven sites are complete and have been accepted by the stakeholders, two sites are complete but require stakeholder acceptance, and seven remaining sites are considered partially done. The project is expected to successfully remediate all eleven required sites by end of FY 2014/15.
Of note, the Contribution Agreement fostered a high level of trust between disparate groups, which has benefitted all stakeholders.
CRS Recommendation
1. The ADM(IE) should ensure that the MNR prepares the completion attestations for all 11 sites by the end of summer 2014, and that the attestations are received by March 31, 2015.
OPI: ADM(IE)
CRS Recommendation
2. As the CA in its current form expires at the end of FY 2014/15, and as the MNR must complete the closure of the remaining seven sites in the summer of 2014 and subsequently prepare the appropriate site closure documentation, the evaluation team recommends extending the CA to March 31, 2016.
OPI: ADM(IE)
CRS Recommendation
1. The ADM(IE) should ensure that the MNR prepares the completion attestations for all 11 sites by the end of summer 2014, and that the attestations are received by March 31, 2015.
Management Action
ADM(IE) will collaborate with the MNR to revise the processes that will be included in the Amendment to the CA, to ensure attestations are completed for the 11 sites and received by 31 March 2015.
OPI: ADM(IE) / DGPR
Target Date: December 2014 for process review
CRS Recommendation
2. As the CA in its current form expires at the end of FY 2014/15, and as the MNR must complete the closure of the remaining seven sites in the summer of 2014 and subsequently prepare the appropriate site closure documents, the evaluation team recommends extending the CA to March 31, 2016.
Management Action
ADM(IE) will seek from the MNR an extension to March 31, 2016, of the current CA. The expected timeline to produce the Amendment to the CA is December 2014.
OPI: ADM(IE) / DGPR
Target Date: December 2014 for Amendment to the CA
Annex B—Evaluation Methodology and Limitations
1.0 Methodology
The evaluation team used multiple lines of evidence and complementary qualitative and quantitative research methods to help ensure the reliability of information and data to support the evaluation findings. The methodology established a consistent approach in collecting and analyzing the data to support the evaluation findings, conclusions and recommendations. Based on the evidence obtained from available sources, the evaluation team reviewed the achievement of the expected outcomes, and the Program’s efficiency and economy, to develop a balanced picture of the relevance and performance of the CA. Information and data were correlated to each evaluation question and corresponding indicators. To ensure the validity of the data captured, the evaluation team used a data triangulation approach. The evaluation team also streamlined the key components of the original G&C logic model, and removed the outcomes considered as indicators. (See the Logic Model in Annex C).
1.1 Overview of Data Collection Methods
The data collection methods were selected based on the data required to address the performance indicators. The evaluation team used the following data collection methods to gather qualitative and quantitative data for each type of operation:
- literature and document review;
- key informant interviews; and
- administrative and financial data reviews.
1.1.1 Documents and Files Review
A preliminary document review was conducted as part of the Planning phase by the evaluation team to garner a foundational understanding of the CA. A comprehensive document review was undertaken as part of the conduct phase by the evaluation team, focusing on the relevance and the performance of the CA activities.
The following documents were reviewed by the evaluation team during the conduct phase:
- program documents:
- Treasury Board submission, the CA annual reports and other activity reports;
- administrative documents:
- CA financial reports, correspondence exchange; and
- accountability documents:
- DND/CAF Reports on Plans and Priorities, DND/CAF Departmental Performance Reports, Canada First Defence Strategy and Speech from the Throne.
The document review was conducted using a customized template organized according to the evaluation questions and indicators.
1.1.2 Key Informant Interviews
Key informant interviews and information sessions scheduled with CA stakeholders, who were directly or indirectly involved in delivering the Program, served as an important source of qualitative information.
Individual interviews were conducted in person with the DND project managers, the DCC project advisor, and the ADM(IE) Comptroller. Telephone interviews were conducted with the MNR staff.
1.1.3 Review of Financial and Administrative Data
The CA financial data was reviewed to determine the degree of efficiency and economy of the Program’s activities and outputs. The data, which covered five years from 2009 to 2013, was extracted from the CA financial reports.
2.0 Limitations
The following table shows the limitations related to the sources and the mitigation strategies applied to them.
Table B-1. Evaluation Limitations and Mitigation Strategies
Table Summary:
This table shows the mitigation strategy for each limitation.
Limitation | Mitigation Strategy |
---|---|
The possibility that the interviewees would provide biased information and only positive stories about their Program. | A comparison was made between interviewees and other people from the same organization or group, and information from other sources (i.e., documents and files). |
The different perspectives from the viewpoints of DND and MNR staff regarding the CA and its impact on remediating the contaminated sites. | Staff were interviewed separately, notes and comments recorded, and then cross-verified. |
Figure C-1. Logic Model for the Remediation of the Former MCL Radar Sites in Ontario.
Text description for Figure C-1:
Inputs are partnerships, government funding allocations, contractors, and materials and services.
Activities using these inputs are analysis and planning of remediation outputs, engagement of interested parties and obtaining their agreement, implementation, and provision of advice, oversight and verification.
Outputs are, firstly, assessments and permits, plans, contracts, and environmental assessment reports. A second output is made up of status meetings, milestone reports, site completion reports, confirmatory sampling, and project summary reports. Management is an additional output, and the last of four groups of outputs is remediation work.
The immediate outcome of these outputs is that remediation is conducted effectively.
This, in turn, leads to two intermediate outcomes, namely, stakeholder acceptance and compliance with the Contribution Agreement (CA). These lead, in turn, to a single ultimate outcome—which is that contaminated lands are remediated to acceptable levels, thereby improving human health and the environment in local areas.
Table Summary:
This table indicates the data collection methods used to assess the evaluation outcomes to determine the Program’s performance (i.e., effectiveness, efficiency and economy).
Outcome | Measure | Indicators | Data Source | Timing of Data Collection | Responsibility for Collection | Sites Completed |
---|---|---|---|---|---|---|
Quality remediation | Compliance with the EAs | EA process completed and registered; signed permits issued by regulators | Status meetings | Receive data before contract award | Prepared by MNR and observed by DND | 11 of 11 |
Use of integrated planning and business cases | Completed management plans as committed to in the Integrated Project Plan and business case (plans updated annually) | Milestone reports and status meetings | Milestone reports submitted on or before mid-July and mid-January annually. | Prepared by MNR and observed by DND | 8 of 11 | |
QA performed | Successful completion of remediation activities at former Mid-Canada Line radar sites in Ontario | Site Completion report | Site Completion report submitted within 90 days of the site being completed. | Prepared by MNR and submitted to and reviewed by DND (DGE/D Env P) | 2 of 11 | |
Application of lessons learned | Application of lessons learned through implementing a QA action list and business planning | Pre- and post-season QA meetings | Ongoing. | MNR, DND participation | 11 of 11 | |
Stakeholder acceptance | Stakeholder verification of the remediation process and results | Stakeholder verification of program participation and acceptance of results | Reports: Site Completion and Project Summary; site visits; status meetings | Site visits, once per year minimum; status meetings, twice per year. | Conduct site visits and will participate in status meetings | 2 of 11 |
Stakeholder satisfaction | Stakeholders’ acceptance of site remediation | Reports: Site Completion and Project Summary; site visits; status meetings | Site visits once per year minimum; status meetings, twice per year. | Conduct site visits and will participate in status meetings | 2 of 11 | |
Compliance with the Contribution Agreement | Adherence to schedule and contract | Application of lessons learned through implementing a QA action list and business planning | Reports: Site Completion and Project Summary | Site Completion Report submitted within 90 days of the site being completed | Prepared by MNR and submitted to and reviewed by DND (DGE/D Env P).MNR, DND participation. | 2 of 11 |
Attestation of project completion | All site completion attestations completed and signed; and confirmatory sampling that the environmental component of site remediation has been completed according to the applicable criteria | Pre- and post-season QA meetings | Summary report submitted within 90 days of the site being completed: end of FY 2014–15; ongoing | Prepared by MNR and submitted to and reviewed by DND (DGE/D Env P). MNR, DND participation | 2 of 11 | |
Value for money of remediation work | Use of competitive contracting | Competitive contracting process | Contracts | Contract issued prior to work commencing | Prepared by MNR and submitted to and reviewed by DND.(ADM(IE) Comptroller, and DGE/D Env P) | 11 of 11 |
Adherence to cost estimates | Cost controls implemented | Cost management process; Milestone reports | Milestone reports submitted on or before mid-July and mid-January annually | Prepared by MNR and submitted to and reviewed by DND.(ADM(IE) Comptroller, and DGE/D Env P) | 11 of 11 | |
Expertise of project management | Progress of project in actual dollars versus budgeted dollars | Milestone reports | Milestone reports submitted on or before mid-July and mid-January annually | Prepared by MNR and submitted to and reviewed by DND.(ADM(IE) Comptroller, and DGE/D Env P) | N/A | |
Cost of overhead and program management | Expenditures compared to planned costs; planned versus the actual implementation schedule | Milestone reports and Site Completion reports | Milestone reports submitted on or before mid-July and mid-January annually | Prepared by MNR and submitted to and reviewed by DND.(ADM(IE) Comptroller, and DGE/D Env P) | 11 of 11 |
Footnote 1 The other two lines were the DEW line along the 69th parallel, and the Pinetree line along the 53rd parallel in the West and the 50th parallel in the East. Unlike the United States-Canada jointly-operated Pinetree line and DEW line, the MCL was funded and operated entirely by the Royal Canadian Air Force.
Footnote 2 As established by the Canadian Council Ministers of the Environment in 1992.
Footnote 3 There were at least 5 environmental reviews and assessments of the 11 sites over the period 1980–2000.
Footnote 4 Canada’s Green Plan (1990); DND Sustainable Development Strategy Iterations 1–4 (1997–2003); Federal Contaminated Sites Action Plan (2004); CRS Audit of Sustainable Development Strategy (2009); Director General Environment (DGE) Program Review (2010); FSDS Implementation (2010–2013); and DES Development (2011–2013).
Footnote 5 Based on a $30 million Contribution. If an estimated total cost of $110 million is used, the cost per site is $10 million—of which the federal government’s share is $2.7 million for each site.
Footnote 6 Estimated spending for FY 2014/15.
Footnote 7 The original schedule called for spending $29,750,000 by end of FY 2013/14.
Page details
- Date modified: