Evaluation of Contributions in Support of the Capital Assistance Program

June 2014

1258-214 (CRS)

Reviewed by CRS in accordance with the Access to Information Act (AIA). Information UNCLASSIFIED.

Acronyms and Abbreviations

Executive Summary

The evaluation of the Contributions in Support of the Capital Assistance Program (CAP) was conducted by Chief Review Services (CRS) to assess the relevance and performance of the CAP, as part of an assessment for funding continuation. The current term of the CAP funding is effective until March 31, 2016.

Program Description

The intent of the CAP is to optimize the DND and the CAF infrastructure by seeking cost-effective partnerships with local authorities (eligible recipients) who have the capacity to operate and maintain the infrastructure and who may also benefit from joint use opportunities. CAP staff identifies need, and selects, defines, implements and manages CAP projects. The Program has been allocated up to $5.4 million in annual Vote 10 expenditures, although the actual annual spending directly attributed to the CAP 10 expenditures averaged $986,000 over the period of fiscal year (FY) 2008/09 to FY 2012/13.

Relevance

The CAP provides a means for DND/CAF to divest non-core infrastructure and improve core infrastructure. The extent of DND/CAF infrastructure in Canada, the current state of that infrastructure, and the consequent risks of operational failure and/or risks to health and safety, mean that the CAP is aligned to DND/CAF priorities and that it continues to be relevant. The CAP was found to be underutilized both in 2009 and during the period of the current CRS evaluation—although it is determined that there is substantive potential for the CAP in the years to come.

Performance

Evidence indicates that the CAP’s anticipated outcomes are being realized, although not to the full extent possible. Through assessing needs, selecting CAP projects, providing funding and managing projects—in collaboration with the affiliated Base/Wing custodians and Level One² (L1) “Sponsors”—the Program continues to establish collaborations with local authorities to upgrade/expand/construct and own and operate the public infrastructure and services that DND/CAF requires.

Through the establishment of Contribution Agreements (CA) with program recipients, the CAP is contributing to the reduction of DND/CAF non-core infrastructure, although funding and resource utilization issues are constraining the achievement of this anticipated outcome. Issues surrounding a lack of business case development and implementation, however, prevent making an accurate determination of the Program’s actual return on investment.

Through CAP projects, the Program contributes to the improvement, creation and/or maintenance of former DND/CAF non-core infrastructure upon transfer to a recipient—and, ultimately, to optimizing the DND/CAF infrastructure inventory. Ultimately, the CAP enhances the quality of life for CAF members and their families, particularly through projects that involve health/safety benefits.

Considering the cost of the CAs in support of the CAP, the benefits to stakeholders, DND/CAF members and their families were found to be economical. However, CAP expenditures have been historically lower than available funding would allow, and have varied considerably over the evaluation period. The current Vote 10 funding level ($5.45 million) may not be sufficient in the future, given the breadth of potential CAP projects identified to date. If a significant uptake of the Program were to occur, the allocated amount would require adjusting.

A critical alternative delivery mechanism proposed by evaluation stakeholders is the introduction of the ADM(IE) Vote 1 budget funding, which would thereby remove the dependency on L1 Sponsors to assign funding and resources to CAP projects.

Key Findings and Recommendations

Finding 1. Relevance—Continued Need

The CAP provides a means for DND/CAF to divest non-core infrastructure. While there are numerous factors that may affect the completion of CAP projects, new project opportunities continue to be identified.

Finding 2. Relevance—Alignment with Federal Roles and Responsibilities

The DND/CAF’s role in supporting the CAP is appropriate, given the extent of DND infrastructure in Canada as well as potential opportunities to divest its non-core infrastructure.

Finding 3. Relevance—Alignment with Government Priorities and DND Strategic Outcomes

Support to CAP is aligned with Government of Canada (GC) priorities and is strongly aligned with DND’s Real Property Acquisition and Disposal strategic outcome.

Finding 4. Immediate Outcome—Collaborations with Local Authorities

Since the 2009 CRS Evaluation, evidence demonstrates that CAP staff has increased program visibility through focused efforts. New opportunities for collaborative projects are now being realized.

Finding 5. Intermediate Outcome—Reduced DND Non-Core Infrastructure

The prioritization of funding and resources often constrains the CAP and impacts timeframes and program effectiveness. Some CAP processes have been modified in order to improve program effectiveness.

Recommendation 1. Allocate Vote 1 funds to the CAP to remove dependency on L1 Sponsors to assign funding and resources from their operational budgets for infrastructure projects.

OPI: ADM(IE)

Finding 6. Intermediate Outcome—Cost-Effective Development and Maintenance of Infrastructure

The actual reduction of operating/capital costs and cost avoidance is difficult to determine without determining the overall cost savings of CAP projects to the DND/CAF. However, for DND/CAF infrastructure and real property upgrades, the Evaluation has determined that the Program does contribute to reducing departmental liability and associated risks.

Recommendation 2. Ensure that the business case for each CAP project includes an analysis to identify the cost savings to DND/CAF in terms of reduced operating/capital costs and cost avoidance.

OPI: ADM(IE)

Finding 7. Intermediate Outcome and Ultimate Outcome—Improved/New/ Maintained Infrastructure and Optimization of DND Infrastructure

The CAP contributes to the improvement, creation and/or maintenance of former DND/CAF infrastructure upon transfer to a recipient. Ultimately, this optimizes the DND/CAF infrastructure inventory.

Finding 8. Ultimate Outcome—Enhanced Quality of Life in the CAF

The CAP contributes to the enhanced quality of life for CAF members and their families, particularly through projects that involve health/safety benefits.

Finding 9. Efficiency of Output Delivery

The CAP expenditures are presently much lower than available CAP funding and vary considerably over the evaluation period.

Recommendation 3. Update the CAP Guide to reflect current program processes.

OPI: ADM(IE)

Finding 10. Adequacy of Progress Toward Achieving Expected Outcomes with Resources Expended

The resources dedicated to the CAP are adequate for the current level of program utilization. If the uptake of the Program were to increase, current staffing and funding levels would not be sufficient.

Recommendation 4. The Terms and Conditions of the Contributions in support of the CAP should be continued after 31 March 2016.

OPI: ADM(IE)

Note: Refer to Annex A—Management Action Plan for a complete list of recommendations and management responses.

1.0 Introduction

1.1 Profile of the Capital Assistance Program

1.1.1 Background

This report presents the findings and recommendations of the evaluation of CAs in support of the CAP. In accordance with Treasury Board’s Policy on Transfer Payments,³ an evaluation is required to support the continuation of the Program’s terms and conditions after March 31, 2016.

CRS conducted a previous evaluation of the CAP in 2009. Where relevant, the current evaluation utilizes those results as a baseline to estimate continuous improvement.

1.1.2 Program Description

The CA between the Treasury Board of Canada Secretariat (TBS) and DND/CAF for the CAP was initially signed in 1983 to provide community-based services and infrastructure to DND/CAF bases and wings. DND/CAF uses the CAP to fund capital projects to:

• upgrade infrastructure after it has been transferred from DND/CAF;
• expand and/or upgrade local infrastructure (i.e., infrastructure that was never owned by DND/CAF); or
• construct new, local, shared-use infrastructure.

As part of the screening checklist,⁴ CAP projects must meet one of the following principal objectives of the Program:

• reduce DND non-core infrastructure and non-core services via consolidation and joint-use opportunities with eligible recipients;
• reduce DND operating, maintenance and recapitalization costs of municipal-type works and public facilities over the long term;
• modernize or construct locally-owned facilities for shared use by DND and local communities; or
• enhance the quality of life in the CAF.

The rationale for the CAs is that local authorities (recipients) are in a more cost effective position to deliver community-type services and maintain non-core infrastructure than DND. CAs also optimize DND infrastructure to support the evolving force structure, allowing for collaboration with recipients and enhanced quality of life of the CAF and their dependents living and working at bases/wings in Canada.

1.1.3 Program Objectives

The overarching objective of the CAP is to optimize DND infrastructure by seeking cost-effective solutions with eligible recipients who have the mandate and the capacity to operate and maintain the infrastructure, and who may also benefit from joint-use opportunities.

The specific activities, outputs and outcomes of the CAP are illustrated in the Logic Model shown in Annex C.

The CAP is primarily involved in assessing needs and selecting, defining, implementing and managing projects. Through the establishment of CAs with eligible recipients, the CAP supports collaborations with local authorities to upgrade/expand/construct and own and operate public infrastructure and services that DND/CAF requires. CAP projects are expected to contribute to the reduction of DND/CAF non-core infrastructure, the cost-effective development and maintenance of infrastructure, and improved/new/maintained infrastructure for use by DND/CAF members and their families. Ultimately, these outcomes are expected to contribute to the optimization of DND/CAF infrastructure to support the evolving force structure, and to enhance quality of life in the CAF.

1.1.4 Program Stakeholders

The key stakeholders of the CAP are eligible recipients and beneficiaries (DND/CAF members and their families). The following groups of recipients are eligible for funding under the CAP:

• provinces;
• territories;
• municipalities;
• provincial and territorial Crown corporations and agencies;
• other corporations owned or controlled by provinces, territories, and municipalities; and
• other organizations that in the normal course of business provide public services (e.g., gas, cable, electricity providers).

DND/CAF stakeholders for this Evaluation are the Deputy Minister, Chief of the Defence Staff and ADM(IE). The Program Manager is the Director Real Property Program Management (DRPPM).

1.2 Evaluation Scope

CRS undertook this evaluation study between September 2013 and March 2014. It was conducted in accordance with the DND/CAF Five-Year Evaluation Plan 2013/14 to 2017/18, and examined the relevance and performance of the CAP for the period 2009 to 2014. This period does not coincide with the program’s funding cycle (FYs 2010/11 to 2015/16), but follows the previous CRS Evaluation conducted in 2009.

CAP program-related activities specified within the Contribution Program include both Vote 1 and Vote 10 expenditures.⁵ The Vote 1 component is funded by the L1 Sponsor for administrative components of activities within the Program; for example, the employment of DND/CAF personnel in support of CAP activities. Vote 1 funds allocated for the Options Analysis Phase⁶ of CAP projects are also funded by the L1 Sponsor. More discussion on this appears throughout the report.

1.2.1 Coverage and Responsibilities

The CAP falls under the 2009 Program Alignment Architecture (PAA) 1.4.1. Real Property Acquisition and Disposal. Approximately 0.21 percent of the total $2.3 billion allocated to this PAA element over the last five fiscal years (FYs 2008/09 to 2012/13) was expended on CAP projects ($4.9 million, not including Vote 1 expenditures).

The CAP Program Manager is responsible for the financial planning and management of CAP, and is accountable to ADM(IE) through DRPPM and the Director General Portfolio Requirements. The CAP Program Manager aligns the CAP Strategy with these four elements: the Department’s Real Property Strategy; program planning and endorsement; program promotion; and monitoring and reporting on the progress and performance of the Program.

Overall co-ordination of a CAP project is the responsibility of the CAP Advisor (ADM (IE) staff) assigned to the project by the CAP Program Manager, and is responsible for overall guidance and co-ordination of project activities. The CAP Advisor coordinates the negotiation of both CAs (#1 and #2) between DND/CAF and the Recipient with a team consisting of the DND Project Manager, a Department of Justice representative and Recipient Representative(s), including legal. A CAP project team is normally comprised of the DND Project Manager, the Wing/Base Property Officer, and Recipient Representative(s).

1.2.2 Resources

Annual spending directly attributed to the CAP averaged $986,000 over the period FYs 2008/09 to 2012/13 (Vote 10, not including Vote 1). Although the maximum allowable amount payable by the CAP is $30 million per recipient annually, the VCDS has set a maximum annual Departmental Program Spending coverage of $5.45 million (2010–2015)⁷ for the Program. This amount may be adjusted depending on program requirements.

1.2.3 Issues and Questions

In accordance with the Treasury Board Directive on the Evaluation Function (2009), the Evaluation addressed five core issues related to relevance and performance. The methodology used to gather evidence in support of the evaluation questions is provided at Annex B.

The issues and questions addressed in the Evaluation are as follows (refer to Annex D for the complete evaluation matrix, which also includes specific indicators and methodologies for each evaluation question).

Relevance

• Continued Need. To what extent does the CAP continue to address a demonstrable need and is responsive to the needs of Canadians?
• Alignment with Federal Roles and Responsibilities. How do the roles and responsibilities of the federal government align with the delivery of CAP?
• Alignment with Federal Government Priorities. How does the CAP align with federal government priorities and departmental strategic outcomes?

Performance (Effectiveness)—Immediate Outcome

• To what extent has the CAP contributed to the establishment of collaborations with local authorities to upgrade/expand/construct and own and operate public infrastructure and services that DND/CAF requires?

Performance (Effectiveness)—Intermediate Outcomes

• To what extent has the CAP contributed to the reduction of DND/CAF non-core infrastructure?
• To what extent has the CAP contributed to the cost-effective development and maintenance of infrastructure?
  • reduced operating/capital costs;
  • cost avoidance; or
  • minimized DND liability.
• To what extent has the CAP contributed to improved/new/maintained infrastructure for use by DND members and their families?

Performance (Effectiveness)—Ultimate Outcomes

• To what extent has the CAP contributed to the optimization of DND/CAF infrastructure to support the evolving force structure?
• To what extent has the CAP contributed to the enhanced quality of life in the CAF?

Performance (Efficiency and Economy)

• Is the DND/CAF delivering programs/services to end users in an efficient manner? What is being spent on the CAP?
• Is the progress made toward expected outcomes adequate for the resources expended?

2.0 Findings and Recommendations

Evaluation findings and recommendations are articulated in Sections 2.1 through 2.6.

The following sections examine the extent to which the CAP continues to address a demonstrable need and is responsive to the needs of Canadians; how well the program aligns with federal roles and responsibilities, including government priorities and DND/CAF strategic outcomes; and the extent to which the program has achieved expected outcomes. Finally, an assessment of the CAP’s performance in terms of efficiency and economy is provided.

To make this determination, the following indicators were assessed:

• analysis of empirical data (including program activities and results); and
• assessment of qualitative evidence provided by documents reviewed, case studies, as well as the opinions of program managers and program stakeholders.

2.1 Continued Need for the CAP

Documents reviewed by the evaluation team and stakeholder interviews led to this finding:

Key Finding 1: The CAP provides a means for DND/CAF to divest non-core infrastructure. While there are numerous factors that may affect the completion of CAP projects, new project opportunities continue to be identified.

Infrastructure provides critical support to DND/CAF operations. It is used to sustain military personnel, provide training facilities, store and protect essential material, and deploy forces. As with many other federal departments, there has been a significant decline in the state of DND’s real property infrastructure in the last twenty years. In fact, 43 percent of its non-residential buildings and 61 percent of works⁸ are over 50 years old. Older infrastructure is typically less suited to serve current needs and, when not maintained in good condition, it carries increased risks of operational failure and/or risks to health and safety. In response to this issue, the 2008 Canada First Defence Strategy (CFDS) commits to spending $40 billion over 20 years, or 8 percent of all defence spending, on maintaining and renewing the Department’s real property. In a discussion of reforms to real property management, the Defence Renewal Charter refers to the CAP as a mechanism to reduce the size of the DND/CAF infrastructure portfolio, and to reduce real property replacement costs, allowing for “reinvestment in higher priority areas to achieve infrastructure commitments.”⁹

A critical task of the current evaluation study was to determine the extent of the CAP’s actual contribution to the reduction of the size of the DND/CAF infrastructure portfolio (and the reduction of real property replacement costs), which will essentially demonstrate the ongoing need for the Program. The continued utilization of the CAP, in terms of the number of CAP projects identified and/or initiated and the number of projects completed, can be an indication of the ongoing need for the Program.

Although the 2009 Evaluation found that the CAP was relevant to federal government priorities, it also determined that it was underutilized. Between 2001 and 2008, eighteen potential projects had been identified and/or initiated, but only three had been completed (with two additional projects remaining in consideration). The current evaluation study found that the CAP has made no significant progress since that Evaluation. In the current evaluation period (FYs 2008/09 to 2013/14) three projects have been completed and seventeen have been identified and/or initiated. These projects fit into four categories, as assigned by CAP staff:

• active (10 projects since 2008);
• inactive (3 projects since 2011);
• closed/incomplete (1 project); and
• closed/completed (3 projects).

These categories are explained in more detail in Section 2.4 of this report.

It is important to note that the CAP anticipates undertaking several new projects in the future. Currently, eight potential projects are identified in program documentation, with a total assigned value of $138 million.¹⁰ This could be indicative of substantive potential for the CAP.

2.2 Alignment with Federal Roles and Responsibilities

This section’s finding is based on a review of documentation.

Key Finding 2: The DND/CAF’s role in supporting the CAP is appropriate, given the extent of DND infrastructure in Canada as well as potential opportunities to divest this infrastructure.

Various levels of government in Canada provide community-based services (i.e., arenas, gymnasiums, water treatment plants and water distribution systems, etc.) and residential accommodation. The development and establishment of collaborative arrangements with CAP recipients¹¹ ensures that DND/CAF can transfer, upgrade and/or develop infrastructure that supports CAF members and their families. Given the extent of property/infrastructure owned by DND/CAF,¹² its role in supporting the CAP is appropriate.

2.3 Alignment with Government Priorities and DND Strategic Outcomes

This section’s finding is based on a review of project case studies and documentation.

Key Finding 3: Support to CAP is aligned with GC priorities and is strongly aligned with DND’s Real Property Acquisition and Disposal strategic outcome.

In the 2013 Speech from the Throne, the government committed to reduce the size and cost of government to ensure taxpayers get value for money. By transferring its non-core infrastructure to local governments, DND supports this priority.¹³ A reduction in the size of DND’s real property portfolio, and reductions in real property replacement costs, will result in reduced requirements for maintenance and repair, allowing DND to reinvest in higher-priority areas to achieve infrastructure commitments. DND infrastructure renewal is expected to generate a re-investment opportunity of between $110 million and $185 million by FY 2017/18. Although the CAP’s contribution to this re-investment has not been identified, the Defence Renewal Charter specifies that the CAP “will be used to the extent possible to transfer non-essential works and real property to neighboring municipalities.”¹⁴

Under the Property and Informatics Infrastructure Acquisition and Disposal Program Activity of DND’s Departmental Performance Report (2011/12) identifies several challenges to DND’s infrastructure portfolio. One of these is to identify new real property portfolio divestment opportunities, as well as to act upon real property divestment in general, in a timely manner. DND is currently focused on the overall real property portfolio so that divestment opportunities can be identified to ensure the affordability and sustainability of the overall Defence portfolio. The CAP essentially contributes to this effort by identifying opportunities and acting on real property divestment.

One CAP project analyzed as an evaluation case study illustrates how project/program objectives align with DND/CAF departmental objectives. The Comox Bicycle Lane project, a Contribution Agreement of $894,000,¹⁵ constructed bicycle lanes along Military Row at Canadian Forces Base (CFB) Comox for use by DND/CAF and surrounding community members. The project contributes to departmental objectives by offering alternative transportation, encouraging physical fitness, and reducing carbon footprint—all which are in keeping with DND sustainability policy.¹⁶ Furthermore, the adoption and maintenance of a healthy lifestyle by CAF personnel is a critical component to success in operations, as described in the Canadian Forces Health and Fitness Physical Strategy.¹⁷

A key objective of ADM(IE) Transformation is to improve delivery of the Capital Construction Program and Projects; to ensure the delivery process is proportional to complexity, cost, urgency, importance and risk. In this context, the CAP aligns with departmental priorities by establishing priorities and implementation strategies in its long-term planning (i.e., five-year plan). CAP projects have been prioritized¹⁸ in the Capital Investment Plan (Infrastructure) 2011–2021.

2.4 Achievement of Expected Outcomes (Effectiveness)

The following sections provide findings as to whether the Contributions to the CAP have achieved the intended results over the course of the evaluation period. The effectiveness of the Contributions to the CAP was assessed by applying appropriate performance measures and/or key performance indicators against each expected outcome. Data for the performance measures was obtained from reports, case studies, documents and financial information provided by program stakeholders. Based upon analysis of this information, the overall effectiveness of the Contributions to the CAP was determined. Accordingly, an assessment was made of the following outcome areas:

Immediate outcomes

• Collaborations with local authorities.

Intermediate outcomes

• Reduced DND non-core infrastructure.
• Cost-effective development and maintenance of infrastructure.
• Improved/new/maintained infrastructure.

Ultimate outcomes

• Optimization of DND infrastructure.
• Enhanced quality of life in the CAF.

2.4.1 Collaborations with Local Authorities

The finding in this section is based on case studies, documentation reviewed and interviews with stakeholders.

Key Finding 4: Since the 2009 CRS Evaluation, evidence demonstrates that CAP staff has increased program visibility through focused efforts. New opportunities for collaborative projects are now being realized.

The 2009 CRS Evaluation found that the limited promotion of the CAP within DND/CAF had contributed to a lack of knowledge and understanding of the Program among internal stakeholders and potential users. Many stakeholders to the current evaluation were unsure of the extent to which the CAP is known and understood among the target stakeholder communities. However, once collaborations with CAP staff were established, many program stakeholders qualified their relationship with CAP staff as good.

In reviewing case studies for the evaluation study, illustrations of how the CAP leverages and strengthens established relationships were discovered. For example, the Business Case of a project that was reviewed¹⁹ states that the CAP is an “excellent opportunity to strengthen the already successful working relationship with the city.” This perception was also expressed by other stakeholders who were consulted.

The 2009 Evaluation recommended that the CAP undertake promotional activities to increase awareness of the program, including by means of revised written documentation, website update and possibly a tour of bases/wings. Since then, program staff continues to undertake these activities. Some initiatives include the creation of a CAP Guide,²⁰ and an internal intranet webpage.²¹ The evaluation was informed that an information booklet is being developed and that an internet website (external to DND/CAF) is planned.

2.4.2 Reduced DND Non-Core Infrastructure

The findings in this section are based on documents reviewed for the evaluation and stakeholder interviews.

Key Finding 5: The prioritization of funding and resources often constrains the CAP and impacts timeframes and program effectiveness. Some CAP processes have been modified in order to improve program effectiveness.

The Defence Renewal Charter outlines the objective to reduce the size of the Defence real property portfolio and reinvestment in high priority areas; to focus on areas critical to supporting modern capabilities.²² A conclusion of the 2009 Evaluation of the CAP was that DND did not have a strategic master plan with a priority list of specific holdings to be transferred. DND has since developed the Capital Investment Plan (Infrastructure) 2013–2023 that articulates a long-term strategy for managing the operations and maintenance or divestiture of its assets. This Capital Investment Plan (Infrastructure) aligns with departmental priorities as outlined in the CFDS and responds to new TBS and departmental requirements.

To measure the contribution of the CAP to reducing DND non-core infrastructure, a baseline measure is important. The Capital Investment Plan (Infrastructure) A-Base²³ program includes 234 projects with a total estimated cost of $4.9 billion.²⁴ Results are expected to be measured against this baseline, whereby success and progress may be tracked by year. In total, thirteen CAP projects are included in this Capital Investment Plan (Infrastructure), with a total value of $293 million, representing 0.06 percent of the Capital Investment Plan (Infrastructure).

In the following table, ten projects categorized as “active” by CAP Staff fall within the current evaluation period (FYs 2008/09 to 2013/14).²⁵ Most of these projects are at the definition phase.²⁶

Table 1. “Active” CAP projects by Year and Assigned Value

Stakeholders consulted for the evaluation indicated that it can take several years from the project start date to the signing of the first contribution agreement (design phase). For example, one project has been “active” since 2005.²⁷ As explained by CAP staff, CAP projects require negotiating with outside parties, and project timelines often change. For example, recipient elections, recipient reorganizing, or a lack of project staff are all factors that can impact a project’s timelines.

A Project Approval Directive²⁸is applied to project approval for the Definition Phase of CAP projects. This Directive explains that the funding for the CAP comes from two sources: Vote 1 for the Options Analysis Phase – funded by the L1 Sponsor, and Vote 10 for the Definition and Implementation phases – funded by ADM(IE) (CAP funding). As pointed out by evaluation stakeholders, the availability of Vote 1 funding from the L1 Sponsor to conduct initial studies may also delay CAP projects. Given this challenge, it is likely that some “active” projects become “inactive” or never reach completion. Likewise, it is possible that inactive projects may never be reopened. Currently only three projects categorized by CAP staff as “inactive” fall within the current evaluation period (FY 2008/09 to 2013/14), as detailed in the following table. However, it is important to note that there are eight projects in total dating back to 2001 that could potentially be re-opened.

Table 2. “Inactive” CAP projects by Year, Assigned Value, and Reasons Why On Hold

The 2009 Evaluation found that a lack of overall program monitoring (and limited lessons learned) contributed to the reoccurrence of issues, in turn leading to project cancellations. At the time, between 2001 and 2006, three projects had been cancelled. Analysis conducted for the current evaluation found several issues could be causing delays/cancellations of CAP projects. In total, a review of CAP project files²⁹ identified various challenges pertaining to the delay or cancellation of CAP projects. Many were related to legal or jurisdictional issues (i.e., police service jurisdiction between military and municipal police forces), but many were related to the fact that key stakeholders/recipients simply withdrew from the project (due to loss of interest). In the current evaluation period (FYs 2008/09 to 2013/14), only one project had been cancelled, and this was due to provincial funding cuts (in British Columbia).³⁰

Some evaluation stakeholders identified process-related challenges and delays impacting the successful reduction of DND non-core infrastructure. Delays were attributed to the complexity of projects (i.e., involving multiple stakeholders with varying interests and priorities), which impacts on implementation delays or the signing of agreements. Currently, L1s are responsible to assign funding and resources for CAP projects, which can delay processes, and thereby affect overall efficiency. Furthermore, it was suggested by some interviewees that CAP projects do not necessarily have operational relevance and, therefore, are not afforded a high priority by the respective L1s.

Case studies that were reviewed also pointed to identified challenges due to the involvement of multiple stakeholders. For example, the Greenwood Clements Park Water System Transfer project took more time than anticipated to progress due to the many stakeholders involved, which included one from each of 14 Wing, National Defence Headquarters, French and English school boards, a partnering utility consultant, and the Nova Scotia Utility and Review Board.³¹

To address these issues, and to improve the effectiveness of the CAP, various initiatives related to the funding, committee organizations and reporting of the program have been undertaken by CAP staff. For instance, CAP staff have sought approval for ADM(IE) Vote 1 budget funding for the development of CAP projects. The proposal to have Vote 1 budget funding within the CAP would conform to the ADM(IE) model of centralized custodianship, and remove the dependency on L1s to assign funding and resources. Ultimately, having a Vote 1 budget within the program would reduce the overall timeframe of projects and enhance the potential for successfully completing projects, ultimately allowing the CAP to utilize its Vote 10 budget more effectively.³²

2.4.3 Intermediate outcome—Cost-Effective Development and Maintenance of Infrastructure

The findings in this section are based on documents reviewed and stakeholder interviews.

Key Finding 6: The actual reduction of operating/capital costs and cost avoidance is difficult to determine without determining the overall cost savings of CAP projects to the DND/CAF. However, for DND/CAF infrastructure and real property upgrades, the Evaluation has determined that the Program does contribute to reducing departmental liability and associated risks.

According to the ADM(IE) L1 Business Plan (FYs 2013/14 to 2016/17), historically there has been insufficient spending on management and repair (M&R) initiatives. The L1 Business Plan reports that M&R spending increased by $44 million, to $281 million in FY 2011/12, which represents 1.14 percent of Real Property Replacement Cost.³³ Although the target of 1.4 percent was not reached, the L1 Business Plan explains that the current level depicts a real growth in M&R expenditures. The L1 Business Plan also explains that the renewed terms and conditions of the CAP have been widely communicated, and there are numerous opportunities for divestments of municipal-type real property holdings that have been identified. Furthermore, it is reported that work continues to progress in negotiations on a number of sites for transfer payments and agreements to be put in place.

The evaluation team sought to examine a program similar to the CAP that assesses the cost of maintenance and operations of federally-owned real property intended for divestiture. Transport Canada’s Port Operations Program was found to be a useful counterpart example. A review of the Evaluation of the Port Operations Program (2012) found a financial model that outlines the cost of operations for Transport Canada if selected Transport Canada properties are not divested. The savings generated for Transport Canada are calculated for the divestment, transfer or demolishing of sites ($22 million between 2007/08 and 2011/12). Based on this, the Port Operations Program offers new owners of the divested infrastructure financial contributions on a “Crown-no-worse-off” model, which considers the value of the property and an estimate of what it would cost Transport Canada to operate it for a specified period.

In light of Transport Canada’s approach, CAP Business Case reports were reviewed, as part of the case study component of the evaluation, to determine whether in the identification of options, the CAP stakeholders compared the cost estimates associated with maintaining the status quo (not transferring property) to the cost of divesting DND real property. Among the case studies examined,³⁴ only one CAP project provided a detailed estimate of the financial impact of the infrastructure transfer. The other two projects examined the benefits of transfer in terms of liability and safety.

In order to truly assess the contribution of the CAP to the cost-effective development and maintenance of DND/CAF infrastructure in terms of reduced operating/capital costs and cost avoidance, it would be necessary for the Program to determine these costs/cost savings for each CAP project (where relevant). Nevertheless, many stakeholders who were consulted perceived that the CAP is ultimately cost-effective in these terms, and strongly agreed that divesting DND/CAF of selected infrastructure ultimately reduced liability and associated risks for the Department.

2.4.4 Intermediate and Ultimate outcomes - Improved/New/Maintained Infrastructure (intermediate) and Optimization of DND Infrastructure (ultimate)

The findings in this section are based on documents reviewed for the evaluation and consultations with evaluation stakeholders.

Key Finding 7: The CAP contributes to the improvement, creation and/or maintenance of former DND/CAF infrastructure upon transfer to a recipient. Ultimately, this optimizes the DND/CAF infrastructure inventory.

Through a review of program files, the evaluation team determined that a greater proportion of projects involves the upgrade of infrastructure after transfer to recipients (64 percent) than either the expansion and/or upgrade of existing local infrastructure (18 percent) or the construction of new local infrastructure (18 percent). The anticipated degree of CAP’s contribution to each of these outcomes has not been defined in program documentation, but the current focus on upgrade of infrastructure after transfer reflects current GC priorities (as discussed in Section 2.3).

All the stakeholders consulted who provided an opinion on this issue felt that the CAP contributes to improved/new/maintained infrastructure (intermediate outcome), and to the optimization of DND’s infrastructure (ultimate outcome).

2.4.5 Ultimate outcome—Quality of life in the CAF

The findings in this section are based on documents reviewed and stakeholder interviews.

Key Finding 8: The CAP contributes to the enhanced quality of life for CAF members and their families, particularly through projects that involve health/safety benefits.

Many stakeholders felt that the CAP does contribute to enhancing the quality of life for CAF members and their families. Some pointed out the health and safety benefits associated with CAP projects; others pointed out that taxpayers are getting more or better service by having the municipalities take over the operations and maintenance of infrastructure, as they have access to more and better resources than DND/CAF in some cases. The reduction of DND/CAF’s liability was proclaimed the most important achievement of the Program by several stakeholders. Ultimately, improvements in safety and reduction in risks benefits members of the CAF and their families, consequently improving their quality of life.

2.5 Demonstration of Efficiency and Economy

The findings in the following sections are based on an assessment of the efficiency of the CAP’s delivery of outputs, what the Program is spending, and whether progress made toward expected outcomes is adequate for the resources expended.

2.5.1 Efficiency of Output Delivery

The finding in this section is based on program information reviewed, stakeholder interviews, and financial information.

Key Finding 9: The CAP expenditures are presently much lower than available CAP funding and vary considerably over the evaluation period.

The following graph presents CAP Vote 10 expenditures and the amounts payable to the Program. The maximum amount payable by the program is $30 million per recipient annually, as determined by the Treasury Board Agreement, but the VCDS approval funding for the CAP is set at $5.45 million per year (total). Spending directly attributed to CAP Vote 10 expenditures averaged $986,000,³⁵ and was unstable over the FY 2008/09 to 2012/13 period. The variation of expenditures reflects challenges experienced by the program, such as delays in the allocation of Vote 1 funding from L1 Sponsors to conduct initial studies prior to project definition (and implementation).

Figure 1. CAP Vote 10 Expenditures. This graph presents the Vote 10 expenditures and amounts payable as per the CA, and the internal cap set by DND for the CAP for FYs 2008/09 to 2012/13. The data is presented in Table 3.

Table 3. CAP Expenditures and Amounts Payable

The CAP experienced particular challenges in FY 2012/13, as reflected by the low expenditures for that year ($50,000), some of which was explained as follows by CAP staff:

• issues related to receiving Vote 1 funding from L1 Sponsors; and
• length of time, after the CA for a CAP project was approved by DND, for the Government of Quebec to issue a decree allowing the project to move forward (14 months).³⁶

Vote 1 expenditures for FY 2013/14 (as of December 2013) were $14,049.37, with a forecast of $60,000 in total for that year. Including the salary-wage envelope for CAP program staff ($213,000), the estimated total Vote 1 expenditures for the CAP is approximately $273,000 yearly. This represents 28 percent in overhead costs.

A CAP initiative leading to greater efficiency of output delivery identified by the evaluation team involves CAP staff dealing with L1s on an individual basis through the Real Property Management Committee (RPMC), in lieu of a specific CAP Working Group. CAP staff explained that using RPMC instead of a CAP Working Group solved a number of challenges, including the effort to streamline and simplify the CAP process, as the L1 representatives at a CAP Working Group were the same as those who attended RPMC.

Another efficiency identified by the evaluation team is that in lieu of producing an annual report, a CAP report is presented to RPMC in the spring and fall of each year (so that it is essentially a bi-annual report). CAP staff explained that this frequency of reporting is appropriate given the nature of CAP projects: many aspects of a CAP project (as previously detailed) are not within the control of the CAP team, and, therefore, regular updates are more prudent. The evaluation team notes that the CAP Guide does not currently reflect these alternative processes and, thus, needs to be updated.

2.5.2 Adequacy of Progress Toward Achieving Expected Outcomes with Resources Expended

The finding in this section is based on the evaluation team’s review of program information and stakeholder interviews.

Key Finding 10: The resources dedicated to the CAP are adequate for the current level of program utilization. If the uptake of the Program were to increase, current staffing and funding levels would not be sufficient.

Most stakeholders felt that, ultimately, the $5.45 million total approved annual funding ceiling is reasonable, relative to the value of defining and implementing CAP projects. In addition, most felt that the progress made toward program outcomes—collaborations with local authorities, reduction of DND non-core infrastructure, and cost-effective development and maintenance of infrastructure—is adequate considering the resources spent on the Program.

Some alternative delivery mechanisms were proposed by stakeholders. The most critical one is related to the introduction of ADM(IE) Vote 1 budget funding to remove the dependency on L1 Sponsors to assign funding and resources to CAP projects.

It was suggested by some stakeholders that the Vote 10 baseline ($5.45 million) on the Program may not be sufficient in the future, given the breadth of potential CAP projects. If significant uptake of the CAP were to occur, the current level of program funding and staffing would be inadequate. However, given the current level of program utilization, coupled with the numerous and unpredictable factors that could impinge on the implementation of CAP projects, it would be imprudent to recommend raising the Program’s current $5.45 million VCDS total approved annual funding at the present time. Should the need for increased annual funding arise, the allocated amount may be adjusted.

2.6 Conclusion

The evaluation team determined that the CAP remains relevant and continues to serve an ongoing need. The Program was found to be well aligned with federal roles and responsibilities, as well as government priorities. Through establishing CA with program stakeholders, the CAP contributes to the establishment of collaborations with local stakeholders. The Program also contributes to the reduction of DND non-core infrastructure, the cost-effective development and maintenance of infrastructure, and improved/new/maintained infrastructure for use by CAF members and their families. Ultimately, the CAP successfully contributes to the optimization of DND infrastructure to support the evolving force structure, and the enhanced quality of life in the CAF.

The CAP has been administered efficiently and indicates a good level of economy for the DND/CAF in consideration of the cost of the Program versus the benefits to the DND/CAF, and members of the CAF and their families. In conclusion, the CAP is effectively achieving all anticipated outcomes, and provides good value for money.

Annex A—Management Action Plan

Management Action Plan

ADM(IE) supports this recommendation. ADM(IE), through ADM(Finance and Corporate Services), will advance the centralization of baseline funding for the development of CAP projects. The target for an initial transfer of baseline Vote 1 funds will be 1 April 2015, as part of a FY 2015/16 business plan.

OPI: ADM(IE)

Target Date: April 2015

Management Action Plan

ADM(IE) agrees with this recommendation. A requirement for analysis and identification of cost savings and cost avoidance for DND/CAF will be included in all CAP documentation, including the Initial CAP Project Screening Checklist and a business case with identified life-cycle cost benefits. The CAP Project Screening Checklist has already been amended, and the business cases will be enhanced by winter 2014.

OPI: ADM(IE)

Target Date: Fall 2014

Management Action Plan

ADM(IE) agrees, and the CAP Guide will be updated to reflect all current program processes by fall 2014.

OPI: ADM(IE)

Target Date: Winter 2014

Management Action Plan

ADM(IE) agrees with this recommendation. ADM(IE) will continue funding the Contributions in support of the CAP.

OPI: ADM(IE)

Target Date: March 2016 and onward

Annex B—Evaluation Methodology and Limitations

1. Methodology

After the initial scoping, a methodology was established to provide multiple lines of evidence, both qualitative and quantitative, in support of the findings on relevance and performance. This methodology was selected to ensure consistency in the collection and analysis of the data addressing performance indicators, as detailed in the Evaluation Matrix based on the Logic Model of the CAP.

1.1. Overview of Data Collection Methods

The Evaluation of the CAP included the use of multiple lines of evidence and complementary research methods as a means to help ensure the reliability of information and data collected. The following data collection methods were used to gather qualitative and quantitative data:

• document review and comparative analysis;
• stakeholder interviews;
• case studies (involving stakeholder interviews and review of project documents); and
• administrative and financial data review.

Where appropriate, the data obtained from lines of evidence was analyzed using a customized template organized according to the evaluation questions and corresponding key performance indicators. Each of these methods is described in more detail in the following sections.

1.1.1 Document Review and Comparative Analysis

Program documentation was reviewed as part of the planning and scoping stage, to identify and build an accurate evaluation matrix (based on the current logic model). GC and DND documents (e.g., Speech from the Throne, CFDS, DND Report on Plans and Priorities, DND Departmental Performance Report) were reviewed as part of the assessment of the relevance of the CAP.

Available documentation on other similar programs within the GC was researched, with the intent to assess best practices and possible alternatives to the CAP. Transport Canada’s Port Operations Program was reviewed by the evaluation team (including an evaluation of this program conducted in 2012).

1.1.2 Stakeholder Interviews

Initial consultations were conducted with program staff members as part of the evaluation scoping. Three program staff members were then interviewed as part of the formal data collection process. In addition, three L 1 representatives of the DND/CAF environments were interviewed, as well a representative of the Canadian Air Division Headquarters.

Interview questions were aligned with the evaluation questions and key performance indicators identified in the evaluation matrix. Interviews lasted approximately one hour, and were conducted by two evaluators in person or by phone, whenever possible. Transcripts were reviewed by two evaluators for quality control, when applicable.

1.1.3 Case Studies

The evaluation team conducted four case studies to provide additional context to the qualitative and quantitative evidence gathered, and to illustrate various challenges and issues with respect to the implementation of CAP projects. The focus of case studies was on CAP projects that have been successfully completed. Six CAP projects in total have been completed since the Program’s inception. Three of these fell out of the evaluation’s scope,³⁷ but since the Admirals/Colville Intersection project was completed relatively recently, it was included.

• Comox—Bicycle Lane (British Columbia, 2012–2013). Construct bicycle lanes at CFB Comox (did not include transfer of real property). Total contribution amount: $894,000.
• Greenwood—Clements Park Water System Transfer (Nova Scotia, 2007–2012). Private Married Quarters demolished; transfer land and municipal infrastructure (water, sewer and road) to the municipality and upgraded to municipal standards. Total contribution amount: $4.5 million.
• Trenton Hwy 2 Entrance (Ontario, 2010–2011). New intersection at entrance to CFB Trenton. Total contribution amount: $320,000.
• Esquimalt—Admirals/Colville Intersection (British Columbia, 2007–2010). Reconfiguration of intersection at entrance to CFB Esquimalt and transfer land to the City of Esquimalt. Total contribution amount: $2.4 million.

Prior to conducting interviews, relevant documentation pertaining to each case study was reviewed (such as business cases, summary reports, etc.), to give context for the interview and to identify gaps. Following the document review, one DND stakeholder and one representative of the province, territory or municipality were interviewed.³⁸

1.1.4 Administrative and Financial Data Review

An analysis was conducted to assess resource utilization (per output) of CAP, which appears under PAA 1.4.1. Real Property Acquisition and Disposal. Consultations were held with CAP program managers to obtain information on the amount of Vote 1 spending to determine the program expenditures for the evaluation period (five years). The data obtained was compiled in an Excel spreadsheet and analyzed by year.

2. Limitations

The evaluation methodology was designed to provide multiple lines of evidence in support of evaluation findings. The data and information were collected to respond to the evaluation questions and issues. As with all evaluations, there are limitations and considerations that should be noted. Thus, evaluation constraints and limitations were considered.

Financial coding used in the Defence Resource Management Information System does not differentiate CAP expenditures from other DRPPM Vote 1 expenditures; therefore, it is not possible to identify specific Vote 1 CAP expenditures. In the new fiscal year, the program Vote 1 expenditures will be in a separate cost centre from DRPPM, and expenses will, therefore, be tracked.

Baseline data was not available, and performance data relevant to performance indicators was limited. A mitigation strategy used was to review documentation and literature, in addition to consulting stakeholders, to derive reasonable baseline standards and measures of performance that could lead to generalized findings. This is expected to provide a realistic baseline from which the evaluation team can measure performance for subsequent evaluations.

Annex C—Logic Model

Figure C-1. Logic Model for CAP. This flowchart depicts the link between CAP activities and outputs, and immediate, intermediate and ultimate outcomes. Note that Vote 1 activities are included as they have a direct impact on program performance. Thus, key management activities are captured in the Evaluation.

Annex D—Evaluation Matrix

Table D-1. Evaluation Matrix—Relevance

Table D-2. Evaluation Matrix—Performance: Achievement of Expected Outcomes (Effectiveness)

Table D-3. Evaluation Matrix—Performance (Efficiency and Economy)

___________________________________________________________________________________________________________________________

Footnote 1 A senior official, either civilian or military, who has direct accountability to the Deputy Minister or the Chief of the Defence Staff.

Footnote 2 A senior official, either civilian or military, who has direct accountability to the Deputy Minister or the Chief of the Defence Staff.

Footnote 3 See http://www.tbs-sct.gc.ca.

Footnote 4 Contained in the CAP Guide (February 2013 draft).

Footnote 5 Departmental resources are received from Treasury Board in the form of Parliamentary Votes. There are three main Votes or funding envelopes received by DND/CAF. These Votes are: a) Operations and Maintenance (Vote 1)—Personnel, Operations and Maintenance funds used for the day-to-day running of departments; b) Corporate Accounts (Vote 1, 5 or 10)—separate funds received to manage some operation(s) on behalf of the Department; c) National Procurement (Vote 1)—separate funds, held at the corporate level by ADM(Mat), used for managing national systems and fleets; d) Personnel (Vote 1)—salaries to pay the Military and/or Civilian personnel; e) Capital (Vote 5)—used to pay building, major equipment, etc.; f) Grants & Contributions (Vote 10)—flow-through funds used to discharge known governmental obligations, such as agreements like NATO infrastructure programs, defence forums and institutions, cadets’ associations, etc.; and g) Statutory—the employer’s non-discretionary share of payroll costs, such as pensions and employment insurance and any other legislated, non-discretionary payments. Source: Vice Chief of the Defence Staff (VCDS)/Defence Management System Manual, Chapter 1, Part 1.

Footnote 6 There are three parts of the process leading to Project Brief Definition approval (Contribution Agreement #1): Options Analysis, Contribution Agreement and Project Brief Definition submission. Source: CAP Guide.

Footnote 7 The annual spending coverage may be adjusted annually, depending on program requirements.

Footnote 8 Works are divided between service works, such as roads, sewers and fuel tanks, and operational works such as airfields, docks, and firing ranges. Source: Report of the Auditor General of Canada to the House of Commons, 2012.

Footnote 9 Defence Renewal Charter, 2013.

Footnote 10 Several of these projects would involve the transfer of Residential Housing Unit Infrastructure. Note that the amounts associated with projects are “placeholder amounts” only, and could be subject to change. Note also that one project does not have a cost associated with it.

Footnote 11 Provinces, territories, municipalities and/or other organizations.

Footnote 12 DND is the single largest property holder in the federal government, owning approximately 21,000 buildings, 13,500 works (including 5,500 kilometers of roads, jetties, training areas, etc.) and 800 parcels of land covering 2.25 million hectares. Source: CFDS, 2008.

Footnote 13 “Our Government will review federal assets; when it is in the best interest of Canadians, they will be sold.” Speech from the Throne, 2013.

Footnote 14 Defence Renewal Charter, 2013.

Footnote 15 To date $49,642 has been spent, as CA #2 has not been completed.

Footnote 16 Described in more detail in Comox Bicycle Lane project Business Case/Options Analysis (2013).

Footnote 17 See www.cg.cfpsa.ca.

Footnote 18 The Capital Investment Plan (Infrastructure) 2011–2021 was produced with the intent of aligning itself with departmental priorities outlined in the CFDS, and to respond to new Treasury Board and departmental requirements for a Departmental Investment Plan. Solid steps have been taken and are planned to re-position this document as the department’s capital construction investment planning tool, to broaden its analyses, assess performance, and to strategically align the Capital Investment Plan (Infrastructure) 2011–2021 with DND/CAF and CFDS strategic investment priorities as reinforced by the National Portfolio Management Plan. Note that CAP projects over $5 million are prioritized in the Capital Investment Plan (Infrastructure) 2011–2021.

Footnote 19 Trenton Highway 2; Business Case, 2010.

Footnote 20 Last updated in 2013.

Footnote 21 See http://admie.ottawa-hull.mil.ca/dgrp/cap-eng.asp.

Footnote 22 Defence Renewal Charter, 2013.

Footnote 23 A-base: A recurring set of funds provided to a department at the onset of each budget period. The base budget is used to keep the department functioning, and is derived from the previous year’s spending and adjustments such as inflation. See www.businessdictionary.com.

Footnote 24 Of the 234 projects, 103 currently have Project Identification approval, 70 have Project Approval (Definition) approval, and 51 have Project Approval (Implementation) approval. The remaining projects are in option analysis. Source: Capital Investment Plan (Infrastructure) Overview.

Footnote 25 There were thirteen projects in total, including those analyzed as part of the 2009 Evaluation.

Footnote 26 All ADM (IE) Construction projects navigate through the standard project framework of Identification, Options Analysis, Definition, Implementation, and Closeout. The Definition phase is defined as “activities leading to the creation of a detailed plan, cost and risk estimate for the implementation of the selected option.” This Phase is also known as Project Brief Definition. Source: ADM(IE) Project Approval Guide; see www.vcds.mil.ca.

Footnote 27 Valcartier Water Connection Project: Valcartier, QC (assigned value $12.8 million).

Footnote 28 The Project Approval Directive provides policy-based direction and guidance. It stems from policies, directives, standards and guidance issued by the TBS, the Privy Council Office, from the Deputy Minister’s Office, and from legislation. Source: Project Approval Directive 2011/12.

Footnote 29 The files were classified as active, inactive or closed.

Footnote 30 Slesse Creek Bridge Upgrade: Chilliwack, British Columbia (assigned value $495,000).

Footnote 31 Greenwood Clements Park: Report to the County Committee of the Whole, 2009.

Footnote 32 In a briefing note to Director General Real Property (6 August, 2013), CAP staff outlines a proposal to seek approval to obtain ADM(IE) Vote 1 budget funding for the development of CAP Projects. In this note, the Vote 1 requirement to advance projects was determined to be $2.5 million over the next 3 to 5 years, with an average budget requirement of $500,000 per fiscal year.

Footnote 33 Real Property Replacement Cost: 1. The current cost to construct a building having the same utility as the subject building but using modern techniques and material. 2. (a) The cost that would be incurred in acquiring an equally desirable substitute property; (b) the cost of reproducing new, on the basis of current prices, a property having a utility equivalent to the one being appraised; it may or may not be the cost of a replica property; (c) the cost of replacing unit parts of a structure to maintain it in its highest economic operating conditions. Sources:1. Real Estate Dictionary; 2. Principles of Right-of-Way Acquisition, International Right of Way.

Footnote 34 One of the CAP case studies was not relevant to this exercise as it involved the creation of new infrastructure (Comox Bicycle Lane), and not the transfer of DND/CAF real property.

Footnote 35 For 3 closed/completed projects in total.

Footnote 36 The project is called Route 369: Valcartier, Quebec.

Footnote 37 Belmont Park Water Systems Upgrade: Victoria, British Columbia (2001); Slip Turn Around Construction Project: Kingston, Ontario (2005), and Admirals/Colville Intersection: Esquimalt, British Columbia (2007).

Footnote 38 Except in the case of the Esquimalt—Admirals/Colville Intersection project, where only two representatives were interviewed due to the unavailability of other stakeholders.

Footnote 39 Involves one of the following: reduced operating/capital costs, cost avoidance, or minimized DND liability.