Evaluation of Infrastructure Management—Real Property

November 2018

1258-3-011 (ADM(RS))

Reviewed by ADM(RS) in accordance with the Access to Information Act. Information UNCLASSIFIED.

Acronyms and Abbreviations

Executive Summary

This report presents the results of the Evaluation of Infrastructure Readiness – Real Property. The evaluation was conducted by the Assistant Deputy Minister (Review Services) (ADM(RS)) in compliance with the Treasury Board (TB) Policy on Results (2016). As the Policy on Results provides flexibility for evaluations to focus on specific program areas of concern or risk, the scope of this “targeted evaluation” examines the Assistant Deputy Minister (Infrastructure and Environment) (ADM(IE)) Group’s immediate outcome of “informed decision making”. The evaluation examines relevance and performance and assesses the achievement of organizational results to inform and support ADM(IE) decision makers in a timely manner during the early stages of the Group’s transformation.

The evaluation assessed the extent that ADM(IE) decision makers and committees have the necessary information to support informed decision making for: real property management; health and safety requirements; environmental management; and obligations regarding indigenous issues. The evaluation also assessed the extent that ADM(IE) performance measures, data, and tools enable - or are being designed to enable - informed decision making across the Headquarters, Regional, and Detachment levels.

Performance

To assess the effectiveness of “informed decision making”, the evaluation focused on three main areas of inquiry: the planning for data; the collection and analysis of data; and the use of data for decision making in the ADM(IE) Group.

Planning for Data: The full capabilities of ADM(IE) data planning and use of data tools continues to be guided by the ADM(IE) Transformation Project. ADM(IE) staff indicated a need for further guidance on the expectations for the use of data and performance measures to support decision making. As accountabilities and authorities continue to be developed, applied and integrated, the improved clarity of roles and responsibilities should also lead to corresponding data planning improvements. For the ADM(IE) environmental and indigenous affairs programs, the absence of clearly defined roles and engagement criteria with other Level One Groups in the DND/CAF has created challenges for ADM(IE) to collect and use this data for decision making and reporting.

Data Collection and Analysis: Some ADM(IE) information management systems are integrated with the Defence Resource Management Information System (DRMIS) and enable staff to analyze data in new ways, however, other systems are still in the process of being integrated. A lack of promulgated data standards and processes has led to issues where data collection requirements are not well understood or coordinated across the ADM(IE) Group. These issues, in addition to reported concerns with the availability and content of training to use the new data management tools, have fostered data reliability concerns among staff which has negatively impacted the ADM(IE) Group’s ability to exploit data and conduct comprehensive analysis.

Use of Data for Decision Making: Within the ADM(IE) Group, relevant available data has been used for project prioritization and integrated resource planning by some divisions and committees to support decision making. Regional and Detachment level staff, however, expressed limited confidence that their data and performance measurement needs will be met through Transformation initiatives. A challenge for ADM(IE) management is to develop a common understanding among all staff of how the Transformation Project is facilitating the increased and improved use of data to inform decisions. The development of a data-centric culture within the ADM(IE) Group would recognize the need for accurate and reliable data for external reporting and the increased use of data in decision making.

Key Findings and Recommendations 

Back to Table of Contents

1.0 Introduction

1.1 Context for the Evaluation

This report presents the findings and recommendations of the Evaluation of Infrastructure Management – Real Property. The ADM(IE) Group transitioned from an advisory function to a nation-wide integrated real property management and service delivery model^{Footnote 1}  through a centralization initiative in 2016, and then embarked on a five-year Transformation Project (2016-2021).^{Footnote 2} This targeted evaluation is conducted in compliance with the TB Policy on Results and directly supports the ongoing ADM(IE) transformation by evaluating the specific outcome area of “informed decision making”.^{Footnote 3} The evaluation assesses the extent that ADM(IE) decision makers have the necessary information and tools, direction and guidance to effectively manage the expanded portfolios and new responsibilities, and is intended to inform and assist ADM(IE) decision makers to further enable success for the ongoing transformation. The evaluation is a component of the FY 2017/18 DND/CAF Evaluation Plan, as approved by the Performance Management and Evaluation Committee (PMEC) in July 2017. The evaluation was initiated in August 2017.

Since 2011, there have been six audits and three evaluations conducted by ADM(RS) and the Office of the Auditor General of Canada related to aspects of the ADM(IE) portfolio.^{Footnote 4}

1.2 Program Profile

1.2.1 Program Description

The mission of ADM(IE) is to enable operational success of the CAF by effectively managing and delivering the Department’s infrastructure, environment and other programs and services to meet the DND/CAF program requirements.^{Footnote 5}  The broader responsibilities of the ADM(IE) Group include:

• Real property services. This includes real property operations, architecture and engineering, construction projects, portfolio planning and procurement services;
• Environmental support and expertise. Effective environmental management is recognized as a critical component to CAF operational readiness, to ensure compliance with relevant laws and regulations, and to sustainably manage the land and infrastructure;
• Contaminated sites management. This includes project management services and provision of expert advice in relation to contaminated sites. This also includes unexploded explosive ordnance issues facing the Department;

• Expertise, advice and coordination on indigenous issues. These activities contribute to ensuring DND/CAF employees are aware of legal commitments and consider indigenous consultation requirements and treaty obligations when carrying out DND/CAF activities, roles and responsibilities;
• A comprehensive nuclear safety program, to ensure regulatory compliance of radioactive materials and sources of ionizing radiation used within DND/CAF; and
• Fire safety and protection services, including fire reporting and investigation, and fire safety engineering.

In support of these responsibilities, the ADM(IE) exercises functional authority^{Footnote 6}  in the following five areas:

• Real Property and immovables lifecycle;
• Environmental protection and stewardship;
• Indigenous Affairs;
• Fire Protection and respiratory protection; and
• Ionizing radiation regulation and safety.

On behalf of the DND/CAF, ADM(IE) is the custodian of the largest real property portfolio in the federal government. These real property assets consist of land, buildings, military-specific operational works and municipal works. Specifically, the DND/CAF maintains approximately 21,000 buildings, including 12,000 residential housing units; 2.2 million hectares of land; and over 13,500 works (such as jetties, airfields, training areas, roads and municipal services).^{Footnote 7} 

The management of the ADM(IE) real property portfolio includes the recapitalization, maintenance and repair of existing buildings, new construction projects, and the disposal and remediation of outdated facilities. Due to the number of DND/CAF locations across Canada, the number and diversity of buildings, the amount of municipal infrastructure for serving CAF members, and various land uses that support CAF operations and training, the management of the portfolio is complex. Real property pressures include the growing proportion of built assets that are at the end of their economic life cycle.^{Footnote 8}  Over half of the infrastructure is more than 50 years old and much of it is not designed to meet modern operational requirements.^{Footnote 9}

1.2.2 Program Background

Prior to 2016, the DND/CAF managed real property under a decentralized model. As the functional authority for real property, ADM(IE) had provided portfolio oversight, management direction and technical advice to nine different CAF custodial organizations across Canada and each organization had direct responsibility for the operations and management of the real property under their control. Since 2011, internal studies conducted by ADM(IE), audits and evaluations by ADM(RS) and the Auditor General of Canada have highlighted challenges^{Footnote 10}  and proposed improvements to the management of DND/CAF real property. As a result, both the Defence Real Property Management Framework and Strategy^{Footnote 11}  were developed in 2013 with the intent, among other objectives, to implement a centralization of DND/CAF real property activities under the governance of ADM(IE).

The centralization of real property management and real property service delivery was completed in April 2016. ADM(IE) was provided with the authority, responsibility and accountability (ARA) to ensure the real property portfolio supported DND/CAF operations in a suitable and sustainable manner.^{Footnote 12}  Following centralization, the ADM(IE) Group embarked on the five-year Transformation Project to evolve the management and delivery of the ADM(IE) Group’s various programs and services. Interviewees stated that the progress of the Project was stalled by numerous gaps and changes in senior management within the ADM(IE) Group. As senior management positions have been filled, Transformation Project initiatives are progressing.

1.2.3 Stakeholders

The principal stakeholder for this evaluation is ADM(IE), as the single portfolio manager and custodian for real property service delivery in the DND/CAF. Other key stakeholders linked to the ADM(IE) portfolio include:

• Commander Royal Canadian Navy
• Commander Canadian Army
• Commander Royal Canadian Air Force
• Assistant Deputy Minister (Materiel)
• Commander Canadian Special Operation Forces Command
• Commander Canadian Joint Operations Command
• Military Personnel Command
• Chief of Force Development
• Public Services and Procurement Canada
• Defence Construction Canada

Back to Table of Contents

1.3 Evaluation Coverage

1.3.1 Scope Background

An evaluation of Defence Real Property was first initiated in January 2016 as a component of the previous five-year DND/CAF evaluation plan. In October 2016, a deferment of the evaluation to the next five-year evaluation plan was accepted by the Departmental Evaluation Committee.^{Footnote 13}  This deferment was based on two premises: potential duplication of findings and Management Action Plan responses resulting from the 2016 ADM(RS) Audit of Municipal Works;^{Footnote 14}  and to allow the opportunity of additional time to review the progress of transformation and the achievement of results.

The scheduling of the current Evaluation of Infrastructure Management – Real Property was approved by PMEC in July 2017, with a plan to evaluate the effectiveness of real property management in the Pacific Region. During initial evaluation scoping activities with ADM(IE) staff it was determined that this evaluation focus would be unrepresentative of key issues and ill-timed with the ongoing transformation.

1.3.2 Evaluation Scope

The revised evaluation scope focuses on the centralized ADM(IE) organization and the achievement of organizational results by examining the specific ADM(IE) immediate outcome area of “informed decision making.” The evaluation assesses the extent that:

• ADM(IE) decision makers and committees have the necessary information to support informed decision making for real property management; health and safety requirements; environmental management; and obligations regarding indigenous issues; and
• ADM(IE) performance measures, data, and tools enable - or are being designed to enable - informed decision making across the Headquarters, Regional, and Detachment levels.

The evaluation recognizes the critical importance of having information that is accurate, timely, relevant, and complete to support effective decision making in the ADM(IE) Group and achieve program outcomes. The use of information for decision making and information management were noted as risks for DND in the 2016/17 Management Accountability Framework and in the ADM(IE) Integrated Resource Plan for 2017/18 respectively.

The process that led to the centralization of the ADM(IE) portfolio is not a focus of the evaluation. The ongoing five-year Transformation Project is also not explicitly evaluated, except to assess the extent that ongoing transformation initiatives support decision making within the ADM(IE) Group. The ADM(IE) Group’s capacity for, and extent of, engagement and decision making with other DND/CAF Level One Groups was not within the scope of the evaluation.

1.3.3 Program Resources

ADM(IE) has an approximate $1.8 billion annual budget and employs approximately 13 percent (2,970) of the DND civilian workforce and 290 military members.^{Footnote 15}

1.3.4 Issues and Questions

In accordance with the TB Policy on Results, the evaluation addresses questions related to program relevance and performance.

• Relevance: The evaluation assesses the extent that ADM(IE) decision making supports Government of Canada priorities. The discussion of relevance focuses on the alignment of ADM(IE) decision making with the TB Policy on Results and Canada’s defence policy.
• Performance - Effectiveness: The evaluation assesses the extent that ADM(IE) decision makers and committees have the necessary information to enable decision making within the ADM(IE) Group in support of ADM(IE) key outcomes.

An evaluation matrix listing each of the evaluation questions, with associated indicators and data sources, is provided in Annex D. The methodology used to gather evidence in support of the evaluation questions can be found in Annex B.

Back to Table of Contents

2.0 Findings and Recommendations

The following sections of relevance and performance discuss the ADM(IE) outcome area of “informed decision making”. The evaluation addresses relevance through the outcome’s alignment with the TB Policy on Results and the 2017 defence policy: Strong, Secure, Engaged. As a measure of performance and to assess the achievement of this outcome, the evaluation looked primarily at the ADM(IE) functional areas of Real Property Management (including health and safety considerations), Environment, and Indigenous Affairs.

2.1 Relevance—Alignment with Government Priorities

The discussion of relevance is based on evidence from document reviews and interviews.

Among questionnaire respondents at the Regional and Detachment levels who chair ADM(IE) committees, more than half stated that committee guidance documents relevant to their work currently lack direction on how data and performance measures are to be used in the committees’ decision-making processes. These same respondents indicated unanimously that they are not aware of any consultation or other process to identify the data and performance measurement needs of their respective committees. These responses may indicate limited progress to update governance documents to include data considerations across the ADM(IE) Group, or a lack of awareness of ongoing initiatives.

Roles and Responsibilities

A key enabler of effective data requirements planning is a clear understanding across the organization of the roles and responsibilities within each functional area and at each level of management. As Transformation Initiative #36, “Define Accountabilities and Authorities,” progresses, an improved clarity of roles and responsibilities for all Level Two and Level Three organizations should lead to corresponding improvements in the approach to data planning. At the time of the evaluation, the completion of Initiative #36 was behind schedule and was reported as “on hold” until the macro-organizational model was reviewed.^{Footnote 22}  Some senior interview participants advised that the parameters of ARA within the ADM(IE) Group are generally defined, but added that additional clarity of roles and responsibilities for ADM(IE) Level One and Level Two staff will be helpful as the ARAs are not well understood, applied consistently or integrated. The ADM(IE) Integrated Resource Planning Directive, however, is contributing to the definition of ARA requirements by identifying the organizational priorities that are to be incorporated into annual Level Two planning. In addition, the ADM(IE) Functional Planning Guidance document articulates the service provision and advisory functions that ADM(IE) delivers to the DND/CAF at large, and identifies lead organizations and points of contact responsible for these functions.

Interview participants noted the absence of clearly defined roles and engagement criteria between ADM(IE) organizations for environmental issues^{Footnote 23}  and for the Indigenous Affairs Secretariat (IAS) and its liaison between other DND/CAF organizations, which have created challenges for the ADM(IE) Group. Interview participants stated the ADM(IE) Group primarily operates within discrete entities or silos, and as such, a shared understanding of the ARAs within the ADM(IE) Group is limited. Furthermore, they opined that a common understanding of roles and responsibilities will improve the stewardship of ADM(IE) portfolios and will contribute to evolving the Group’s focus from transactional activities to more strategic decision making.

Information Management Tools

Evidence from interviews and the document review showed that the full capabilities of the ADM(IE) data collection tools continue to be developed as the scope of data requirements is adjusted, user-interface issues are managed and new, emerging data needs are identified.

For real property management, information management systems and processes are being refined to meet the needs of stakeholders, including the user communities at the Real Property Operations Regions and Detachments. The IE Business Modernization (IEBM) project, completed in March 2017, was intended to support modern business governance through the implementation of integrated and standardized business processes and a robust information management solution, based on industry best practices.^{Footnote 24}  It was designed to replace and consolidate the functionality of former obsolete applications used to track maintenance, life safety and code compliance for real property into DRMIS. The IEBM project resulted in the introduction of new data management and analysis modules available through DRMIS, referred to as the Infrastructure and Environment Resource Information System (IERIS).

Roles, Responsibilities, and Business Processes

An important factor contributing to the success of the ADM(IE) outcome area of informed decision making is a clear understanding, across the organization, of the roles and responsibilities for data collection at each level and within each functional area. By the nature of their position in Regional and Detachment offices, the Real Property Operations staff is often responsible for the collection of information and data for other ADM(IE) Divisions. Inconsistent identification of data collection requirements for Real Property Operations staff, however, has sometimes led to issues where data collection requirements are not well understood or coordinated. Questionnaire responses from Regional and Detachment Commanders suggest there are opportunities to improve the communication of data standards and processes to achieve a consistent approach across the organization. The questionnaire respondents also described varying and inconsistent approaches to address gaps where required data and performance measures were found to be unavailable or inadequate. Examples of actions taken range from escalating issues through the appropriate chain of command to trying to address issues locally. The 2016 Audit of Municipal Works also noted the need for the ADM(IE) Group to implement information management processes. To ensure the success of the ADM(IE) data collection model, common direction and standards should be applied to the decentralized data collection activities across the organization, to ensure information is available, collected and recorded in a consistent manner across different regions, detachments and functional areas.

Organizational issues

Capacity and capability to conduct analysis

Interview participants reported that while their staff capacity is presently sufficient to analyze currently available, albeit limited, levels of reliable data, exploiting new data sources and emerging analysis capabilities through DRMIS may require additional staff capacity. On the other hand, one interviewee commented that the increased use of DRMIS for automated data analysis could potentially reduce the demand on staff. A significant number of Regional and Detachment questionnaire respondents indicated that they did not have sufficient time or available personnel to conduct data analysis. Several respondents stated they have not taken any action to address unavailable or inadequate data and performance measures, citing limited human resources that cannot be taken from day-to-day responsibilities to address systemic data issues. In addition, limitations in DRMIS/IERIS that allow only certain personnel to use the system to perform certain functions (known as role assignments) have also had an impact on the ability of staff at Regions and Detachments to use data from IERIS. An interview participant explained that role assignments have a large impact on smaller detachments where there are limited staff and many roles are required to be assigned; one individual is unable to hold more than one role assignment, limiting their ability to effectively utilize DRMIS.

The skill level of ADM(IE) staff is also impacting the ability to exploit new DRMIS capabilities. Multiple interview participants reported that additional training would improve the use of DRMIS for data analysis. In fact, some staff indicated that they have not yet received sufficient training to leverage the full capabilities of DRMIS to address the obligations associated with regulatory and compliance-based reporting. Of the training that was received, one interview participant commented that the training focused on basic use of the IERIS interface rather than “the why” or leveraging the capabilities of the system. For some ADM(IE) staff, skill fade has been an issue; as the use of IERIS may be infrequent, resulting in the loss of skills gained in initial training. Training appears to be an issue at the Regional and Detachment level as well, given that 20 percent of questionnaire respondents stated that the limited IERIS training was impacting their ability to use the system for data analysis. Questionnaire respondents at the Detachment level noted that because of these limitations, data was not being used to its full potential across all functional areas.

Plans have been developed to improve the capacity of the ADM(IE) group to facilitate the collection and analysis of data over time through the ADM(IE) Human Resources Management Strategy^{Footnote 30}  and ongoing IEBM/DRMIS support from the IE Business and Data Support group as the Transformation Project progresses. Transformation initiatives #2 (adopt IEBM technology enablement), #3 (Develop broader ADM(IE) technology strategy), and #4 (ADM(IE) technology improvements), relating to the adoption of IERIS, all touch on this issue of capacity, from improving/fine-tuning the functionality of the tools and addressing capability gaps, to providing additional training to users of IERIS in the National Capital Region as well as to Real Property Operations Group Regions and Detachments. Interviewees commented that the greater use of data and analysis will take time and, as users spend more time working with the new DRMIS tools, they will become increasingly comfortable with them and learn to use them to their full extent.

Integration of DRMIS and other tools 

Successes and areas for improvement

Despite the range of challenges identified earlier in this evaluation report, interview participants and questionnaire respondents advised that limited, reliable data is available within ADM(IE) Divisions and has been used to inform decisions within functional areas. Within some ADM(IE) Divisions, relevant available data is being used for project prioritization and integrated resource planning. For example, both the Real Property Operations and the Director General Portfolio Requirements Groups use a scorecard system to prioritize Real Property projects, assigning points to projects based on a variety of measures with data extracted from DRMIS and other databases. Interview participants indicated that both Real Property Operations and Director General Portfolio Requirements staff plan to rely on DRMIS to a greater extent as relevant data becomes more available and mature. The interviewees also advised that use of data for decision making is improving as performance measures are now required to support requests for resources, in accordance with the integrated resource planning process.^{Footnote 37} 

Interview participants discussed examples of ADM(IE) committees and sub-committees which have used performance data to effectively support decision making, such as the IE Resource Management Board, the IE Management Committee, the Defence Environmental Management Committee and the IE Board.^{Footnote 38}  While other ADM(IE) committees and sub-committees may also use various types of available data to support decision making, the terms of reference for committees and sub-committees do not explicitly outline expectations for the use of data to support decisions. In addition, the expectation and type of support to ADM(IE) committees by established ADM(IE) information-sharing networks, such as the Environment Network, Indigenous Affairs Network (IAN), Real Property Network and Portfolio Network, have also not been well defined. Interview participants reported that there are no established means to share information outside of the IAN, or for the IAN and the IAS to receive information from Real Property Regions and Detachments and from other Groups within the DND/CAF. Interview participants indicated, however, that additional resources had been acquired to address this issue and develop a more systematic approach to internal information sharing and communications. In addition, interviewees pointed out that there is presently no dedicated comprehensive body, such as an Indigenous Committee or Working Group, to consider all available information on indigenous issues and take decisions on key issues.^{Footnote 39}

Impact of available data

Data availability and reliability directly affect the analysis of data and its use to support decision making. Interview participants advised that, at present, ADM(IE) financial data is the most reliable and available data source, and as such, it is used extensively. Questionnaire respondents also indicated they rely heavily on financial data for real property management decisions at the Regional and Detachment levels. Anecdotal, experiential, and “gut feeling” information was also mentioned as a component of decision making by interviewees and questionnaire respondents. However, in addition to financial data and important contextual, qualitative information gained from experience and expertise, multiple interview participants reported that they would eventually like to incorporate more diverse data sets in order to support well-informed planning and management decisions. Questionnaire respondents expressed mixed opinions in terms of the extent that currently available data sources were useful and supported real property decision making at the Regional and Detachment levels. A significant number of questionnaire respondents reported that, while they are currently using a wide range of data and information sources, they would like to improve data usage and adopt a systematic use of data to support decision making.

Going forward

A continuing challenge for ADM(IE) management is to facilitate an improved awareness and a common understanding across the ADM(IE) Group with respect to the aims and objectives of the Transformation Project. Confidence in the ability of the Transformation Project to address the use of data in decision making differed significantly between ADM(IE) staff at the headquarters level and the Regional and Detachment Commanding Officers within the Real Property Operations Group.

Many interview participants at the headquarters level described their vision for future decision making within the ADM(IE) Group and recognized that the Transformation Project continues to be the major driver for the increased and improved use of data to enable informed decisions. Over three quarters of questionnaire respondents from Regions and Detachments, however, expressed limited confidence that the Transformation Project would be successful and indicated that they somewhat disagreed, strongly disagreed or were neutral that their data and performance measurement needs can be met in the next five years through Transformation. One interview participant noted that for the 39 transformation initiatives to be implemented in a cascading fashion, there needs to be a shared and common understanding of the initiatives, their priorities, and the intended end-state for the ADM(IE) Group.

Several interview participants noted that there continues to be a misunderstanding of centralization versus transformation and the associated objectives among various staff at different levels. The lessons learned section of the 2017 Transformation Year-end Report^{Footnote 40}  stated that “ADM(IE) staff do not have a clear picture of what transformation is and what it means for them.” As an action plan for this issue, the Report indicated the intention to “develop plain language artifacts to explain transformation.” Four of five items listed in the ADM(IE) Transformation Risk Register^{Footnote 41}  noted risks related to change management, organizational capacity, and project management. The Risk Register also noted the impact of each risk and an Action Plan to mitigate the risks.

Moreover, questionnaire respondents highlighted a need for a culture change within the ADM(IE) Group, appropriate resources and meaningful direction in order to meet an end-state that is not yet clearly understood. Interview participants also opined that a broad culture change is required within the ADM(IE) Group to enable a greater use of data to support decision making. The development of a data-centric culture within the ADM(IE) Group would ensure the systematic use of data in external reporting, increased use of data in decision making, and the recognition of the need for accurate and reliable data for these purposes. A key challenge to the success of an emerging ADM(IE) data-centric organization is building both understanding and consensus for the use of data by management, staff, and by committees to support decision making to achieve results.

Management Action

Agreed. To ensure a common understanding and regular updates of the Transformation project, ADM(IE) will develop and execute a Communications Plan to engage stakeholders and all ADM(IE) staff on the progress of the Transformation initiatives and how these initiatives contribute to the desired ADM(IE) end-state.

Back to Table of Contents

Annex B—Evaluation Methodology and Limitations

1.0 Methodology

The evaluation of the Infrastructure Management – Real Property considered multiple lines of evidence to assess relevance and performance. The aim of this evaluation was to assess whether decision makers have the necessary information available to make well-informed decisions. The evaluation did not assess the effectiveness or the outcomes of decisions.

The methodology established a consistent approach in the collection and analysis of data to help ensure the reliability of the evaluation process. Qualitative information was used to establish context for decision making in the ADM(IE) Group as well as to interpret the significance of key information. Qualitative assessments were used to validate the overall analysis and to develop the evaluation findings and recommendations.

1.1 Overview of Data Collection Methods

Data collection methods were selected based on the data required to address performance indicators in the evaluation matrix. The following data collection methods were used to gather qualitative and quantitative data:

• document and literature reviews;
• stakeholder questionnaire; and
• key informant interviews.

1.2 Details on Data Collection Methods

1.2.1 Document and Literature Review

A preliminary document review was conducted, as part of the planning phase of the evaluation, to garner a foundational understanding of the ADM(IE) organization, and ADM(IE) centralization and transformation initiatives. A comprehensive document review was conducted as part of the conduct phase of the evaluation, focusing on aspects of informed decision making. The literature review was limited due to time constraints.

The following documents were reviewed during the scoping and conduct phase of the evaluation:

• Canada’s defence policy: Strong, Secure, Engaged;
• Corporate documents: Reports on Plans and Priorities, Departmental Performance Reports, Departmental directives, ADM(IE) Strategy, ADM(IE) Integrated Resource Plans, Strategic Planning Guidance, and Transformation documents;
• Legal and policy documents: Acts and Regulations, and TB policies;
• Academic papers and articles; and
• Other documents: ADM(RS) evaluations and audits, Office of the Auditor General audits, other internal studies or reviews.

1.2.2 Stakeholder Questionnaire

A bilingual (English and French) stakeholder questionnaire was developed by ADM(RS) and distributed to Commanding Officers of Real Property Operations Group Detachments and Regions. Recipients were identified using information from Real Property Operations Group and the Microsoft Outlook Address Book/Global Address List (GAL). The survey was distributed to 34 recipients, was online for three weeks, and had a response rate of 94 percent.

The questionnaire asked questions on the subject of whether ADM(IE) decision makers and committees have the necessary data and information to enable well-informed decisions that support results. Questions were organized to receive feedback on data and information specific to the functional areas – Real Property Management (including health and safety), Environment, and Indigenous Affairs – as well as decision making in general, and for ADM(IE) committees.

1.2.3 Key Informant Interviews

Key informant interviews and discussions with ADM(IE) staff and other stakeholders provided important sources of qualitative information.

Informal pre-scoping interviews were conducted with seven key ADM(IE) stakeholders to determine appropriate areas of inquiry within the initially prescribed scope and develop the foundation for the development of comprehensive key informant interview guides. These initial interviews, with additional research, determined that the initial scope was unrepresentative of key issues and risk areas and that the ongoing ADM(IE) Transformation Project had already identified key issues that were being addressed through transformation initiatives.

The revised scope, which focuses specifically on the ADM(IE) immediate outcome area of “informed decision making” was approved in mid-October 2017. To support this area of inquiry, 19 interviews were conducted with 29 ADM(IE) staff at the Director General, Director and staff level. Interviews were semi-structured and conducted in-person, either one-on-one or in a group setting. Some informants were interviewed multiple times. Detailed notes were taken during the interviews and then transcribed and analyzed according to research themes. Interview notes were shared with key informants for fact verification when required. When necessary, interviewees provided written responses to the interview questions after the interview was completed to provide more detailed responses.

2.0 Limitations

Table B-1 describes the limitations and mitigation strategies employed in the Evaluation of Infrastructure Management – Real Property:

Back to Table of Contents

Annex C—Logic Model

Back to Table of Contents

Annex D—Evaluation Matrix

Back to Table of Contents