Evaluation of the Land Equipment Program

1.1 Profile of the LEP

1.1.1 Background

This evaluation presents the findings and recommendations of the DND LEP. The evaluation examined the relevance and performance of this program over the period of 2009 to 2013, and was conducted by ADM(RS) in accordance with the 2009 TBS Policy on Evaluation. This program has not been previously evaluated.

In the conduct of the evaluation, ADM(RS) was supported by an advisory panel comprised of representatives from ADM(Mat). The advisory panel was consulted at key intervals throughout the evaluation, specifically when defining the project scope, developing the logic model, identifying KPIs, and reviewing preliminary findings. The findings and recommendations in this evaluation may be used to inform management decisions related to program delivery and resource allocation and will serve as a baseline for follow-on evaluations.

1.1.2 Program Description

The Materiel Group, under ADM(Mat), is the central service provider and program authority for DND/CAF materiel. The ADM(Mat) is accountable to the Deputy Minister for the full life cycle of materiel, from acquisition, through maintenance and support, to disposal.¹

Within ADM(Mat), DGLEPM delivers and manages land and common materiel required for training and CAF operations. The program supports the life cycle management of approximately 30 major fleets, representing approximately 22,685 platforms and 280,531 centrally managed line items including ammunition, and the management of multiple service and acquisition contracts.

DGLEPM is divided into nine directorates and two field units. Director Land Equipment Program Staff (DLEPS) oversees the business planning and stewardship of financial and human resources within DGLEPM. The Director Land Procurement, the Quality Engineering Test Establishment (QETE), and the 202 Workshop Depot (202 Wksp Dep), provide specialized or unique support to the other directorates in DGLEPM.

Seven directorates within DGLEPM specialize in the life cycle management of specific equipment fleets. These seven directorates are commonly referred to as the “technical directorates.” They are responsible for the conduct of core DGLEPM materiel acquisition and support activities. The technical directorates comprise multiple equipment management teams (EMT), which themselves comprise life cycle material managers (LCMM), technical authorities (TA), supply managers (SM), and a procurement officer (PO), under the management of a section head. The EMTs are directly responsible for the “delivery, maintenance and disposal of CF equipment and systems in the most efficient and cost effective manner possible.”²

1.1.3 Program Objectives

The objective of the LEP is to ensure that a sufficient quantity and quality of land and common equipment and materiel is available for the CAF to meet the readiness requirements of the CA and the CAF with respect to the employment of multi-purpose, combat-capable land forces.³

The specific outcomes of the program in support of this objective are depicted in the LEP Logic Model (Annex C).

1.1.4 Stakeholders

The life cycle management of defence materiel is a whole-of-government activity with multiple stakeholders and complex processes of which DND is not the sole owner.⁴

Stakeholders include, but are not limited to, the following:

• CA (the primary client of the program);
• Canadian Materiel Support Group;
• Royal Canadian Air Force (RCAF) (client for common use items and ammunition);
• Royal Canadian Navy (RCN) (client for common use items and ammunition);
• Canadian Special Operation Forces Command (CANSOFCOM) (client for equipment, common use items, and ammunition);
• Other DND/CAF organizations such as Canadian Joint Operations Command (CJOC) and Strategic Joint Staff;
• Other government departments, such as Public Works and Government Services Canada (PWGSC), TBS, and Industry Canada; and
• Canadian defence industry.

1.2 Evaluation Scope

1.2.1 Coverage and Responsibilities

Based on the 2010 DND Program Activity Architecture, LEP activities are part of sub-sub-program 2.2.4.3 Land Equipment Maintenance.⁵

The evaluation examined the life cycle materiel management activities conducted by the LEP to assess the ongoing need, roles, and responsibilities of the program, the achievement of expected outcomes, and the Program’s overall efficiency and economy.

1.2.2 Exclusions

The evaluation excluded LEP activities that have been evaluated or audited by ADM(RS) or by the Office of the Auditor General of Canada (OAG) prior to 2009. Additionally, activities that will be the subject of a future separate evaluation(s) are not covered in this evaluation. The evaluation scope excluded the following components of the LEP:

• major and minor capital procurement (Vote 5 expenditures);⁶
• national warehousing and storage;⁷ and
• third line maintenance activities related to readiness covered by other evaluations.⁸

1.2.3 Resources

Financial

Between 2009 and 2013, the program expenditures averaged approximately $1 billion per year, although they declined 12% overall during this period. The LEP expenditures represent approximately 5.7 percent of overall expenditures by DND from FY 2008/09 to FY 2012/13. These expenditures, which included contractor service contracts, were primarily for replacement components, minor upgrades, and repairs to land and common equipment. Table 1 presents the LEP National Procurement (NP) expenditures throughout the evaluation period.

Table 1. DGLEPM NP Funding for FY 2008/09 to 2012/13. This table summarizes the LEP Vote 1 expenditures and the percentage variation for FYs 2008/09 to 2012/13.

Personnel

DGLEPM is staffed with civilian, military, and contracted personnel. Table 2 presents the distribution of the military and civilian staff for the period covered by the evaluation (FY 2008/09 to FY 2012/13). The data is based on extracts from HRMS – PeopleSoft. These figures are population,⁹ not FTEs, and exclude personnel in positions dedicated to capability acquisition. The number of DGLEPM contactors has not been formally tracked; however, the expenditures for professional services within the DGLEPM have increased from $3 million to $16 million during the evaluation period.

Table 2. DGLEPM Manning for FY 2008/09 to FY 2012/13.¹¹ This table summarizes the distribution of the military and civilian staff and contracted support for the period covered by the evaluation.

1.2.4 Issues and Questions

In accordance with the TBS Directive on the Evaluation Function (2009),¹² the evaluation addresses the five core issues related to relevance and performance. An evaluation matrix listing each of the evaluation questions, with associated indicators and data sources, is provided at Annex D. The methodology used to gather evidence in support of the evaluation questions can be found at Annex B.

2.0 Findings and Recommendations

The following sections examine the extent to which the LEP addresses a demonstrable need, is appropriate to the role of the federal government, is aligned with DND/CAF priorities, and demonstrates efficiency and economy. To make this determination, the evaluation analyzed program documents, empirical data, and qualitative evidence.

2.1 Relevance—Continued Need

This section examines whether the sustainment of land and common defence materiel continues to address an ongoing demand for the outcomes and objectives of the program. The findings in this section are based on documents reviewed and key informant interviews, with representatives from ADM(Mat) and the CA. The indicator “evidence of current and future need for the program” was used in the assessment of alignment with federal roles and responsibilities.

Key Finding 1: The LEP, as a sustainer of defence materiel, is a critical support element that contributes to force readiness, allowing the DND/CAF to respond to the needs of the GC.

In DND, the life cycle management of materiel has an important and direct link to the readiness of the CAF as materiel must be available in the right quantity, mix, and condition to meet the readiness posture of the CAF.¹³

The LEP is the only program existing within DND that has delegated responsibility to manage land and common materiel. In 2012, DGLEPM, through the LEP, supported the life cycle management of approximately 30 major fleets, representing approximately 22,685 platforms. In addition, the program manages 280,531 centrally managed line items, including all ammunitions, and approximately 100 in-service support contracts with a potential value of over $20 million each.¹⁴ This large portfolio of vehicles and equipment requires ongoing maintenance and repair, as well as upgrades and eventual disposal. Given its size and complexity there has been demonstrable need and manage the demands through a program such as the LEP.

Materiel sustainment demands have increased over the past five years and, going forward, are expected to continue to increase with the acquisition of new and modernized equipment fleets as part of the Canada First Defence Strategy (CFDS).¹⁵ The Defence Renewal Committee recently stated that, “There is also a growing expectation that readiness levels of fleets must be maximized in order to meet operational demands in a security environment that continues to be unpredictable and volatile.”¹⁶

2.2 Relevance—Alignment with Federal Roles and Responsibilities

This section examines the extent to which the program aligns with departmental and federal roles and responsibilities. The following indicators were used to assess the alignment with federal roles and responsibilities:

• alignment of materiel management with government acts and legislation; and
• alignment of materiel management with government policies and strategies.

Key Finding 2: The LEP aligns with the roles and responsibilities set by the federal government.

The LEP clearly aligns with the roles and responsibilities of DND. According to the TBS Policy on Management of Materiel (2006), ¹⁷ federal ministers are accountable for the management of materiel and for the sound stewardship of the materiel entrusted to their organization. In fulfilling the delegated responsibilities to procure and sustain defence materiel, DND is guided by this policy. All government departments are required to manage materiel over its entire life cycle to ensure that both operational requirements and value for money are met when planning, acquiring, sustaining, and disposing of materiel assets.

In addition, The Defence Production Act ¹⁸ assigns the Minister of PWGSC the responsibility for the management of “stockpiled” defence materiel.¹⁹ PWGSC and DND agreed, in principle, to a division of responsibilities between the two departments for the quality assurance of materiel and services acquired. Subsequently, the Minister of PWGSC delegated the authority to purchase defence materiel and services, under the conditions listed in the Procurement Administration Manual, to the Minister of National Defence. ²⁰

2.3 Relevance—Alignment with Government Priorities

This section examines whether the objectives of the LEP are consistent with current GC and DND/CAF priorities. The following indicators were used to assess the alignment with federal priorities:

• alignment with GC defence priorities; and
• alignment with DND/CAF priorities.

Key Finding 3: The LEP is aligned with GC defence priorities.

As stated in the Speech from the Throne (2013), the defence of Canada and the security of Canadians continue to be a top priority for the GC. The GC is committed to ensuring that the CAF has the tools it needs to deal with the full range of threats and challenges to Canada and Canadians.

Through the CFDS, the GC is committed to providing the CAF with the adequate resources for training, spare parts, and the equipment needed to provide effective military support, as well as to modernize and upgrade existing fleets in order to support CAF readiness and training requirements.²¹ This is accomplished by maintaining rigorous stewardship over all assets, including materiel.²² The LEP contributes directly to the achievement of this important priority through the material acquisition and support of land and common defence materiel.

Key Finding 4: The LEP is aligned with DND/CAF priorities.

The DND Report on Plans and Priorities 2013-14, states that a key priority for DND is to “ensure that resources are aligned and available to support determined readiness levels and therefore able to posture military capability to meet planned and anticipated requirements of the Government of Canada.” The Report also states that readiness encompasses the resources needed to maintain equipment.

In delivering the LEP, DGLEPM supports CAF priorities by meeting the ADM(Mat) strategic objectives of ensuring defence affordability, maintaining required CAF Force Posture and Readiness, and strengthening the defence team.²³

2.4 Performance—Achievement of Expected Outcomes (Effectiveness)

Materiel availability is an essential component of CAF readiness.²⁴ To determine the overall effectiveness of the LEP, a logic model was developed (Annex C) that links the cross-functional business processes to the various outputs, and then associates the outputs to anticipated outcomes (program expectations and objectives).

The effectiveness of the program was then assessed by applying appropriate performance measures and KPIs against each outcome. Data required to assess performance was obtained from various sources, including ADM(Mat) and DGLEPM business plans, activity reports, other program documentation, comparative research, an employee questionnaire conducted by the evaluation team, and a project focus group. Additionally, numerous one-on-one interviews were conducted with program managers, program staff, and various stakeholders primarily from the CA as the main client of the LEP program.

Accordingly, an assessment of the LEP was conducted based on the following immediate outcomes:

• provision of sound advice to stakeholders;
• ability to meet the priorities and requirements of the CA and common equipment users;
• disposal of equipment and materials in compliance with legislation; and

• ensuring land and common equipment is safely stored and operated.

2.4.1 Immediate Outcome 1: Provision of sound advice to stakeholders

A key role of the LEP is to provide sound advice in support of the management of the fleets of the CA. This information allows the CA to maximize vehicle availability and reliability, address operating issues and concerns, control expenditures, manage suppliers and contractors, and make overall decisions on the life cycle of the equipment. As such, the advice must be technically sound and timely, and it must provide added value.

The indicators used to assess this immediate outcome are as follows:

• the frequency with which advice is sought and provided;
• the provision of relevant, credible, and timely advice; and
• the utilization of information management planning and decision support tools.

Frequency with which Advice is Sought and Provided

Key Finding 5: The lack of availability and reliability of data limits DGLEPM’s ability to effectively and efficiently track and manage the provision of advice.

One indicator used to assess the effectiveness of the technical advice provided by DGLEPM is the trend line of the frequency of advice provided over time. The logic of the indicator is that, should the advice that the program provides not be of value, over time stakeholders such as CA staff would bypass DGLEPM to find the information they needed elsewhere.

Table 3 includes a summary of the types and frequency of advice provided by DGLEPM. In general terms, interviews with DGLEPM noted that there is regular daily or weekly contact with customers, indicating that advice is in high demand. However, there is no detailed, formally recorded information regarding advice volumes or timeliness. The exception is the Address Indicating Group messages, where between 2012 and 2014 there were 51 Address Indicating Group messages issued. These messages, from EMTs via DGLEPM, give safety advice or specific fleet direction to equipment operators.

Table 3. Frequency of Advice Provided by DGLEPM.²⁵ This table summarizes the types and frequency of advice provided by DGLEPM.

The evaluation team was unable to empirically determine if there has been any change in the frequency with which advice has been sought by stakeholders. However, results from the LCMM/Supply Manager (SM) questionnaire for this evaluation indicated that LCMMs and SMs spend on average 16% of their time on the provision of advice, which is equivalent to the time they report spending on other core activities such as procurement and equipment sustainment. A significant number of LCMMs (20%) indicated that they spend more than 50% of their time providing advice.

Given the amount of resources spent on the provision of advice and its importance to the organization, the DGLEPM Performance Measurement Strategy Framework should include the provision of advice to enable more proactive management and reveal opportunities for efficiency.

An example of an organization within DGLEPM, where performance measurement of information requests has proven to be beneficial, is QETE. The tool used is the Quality Project Tracking System. Requests are tracked via a tasking form that is filled out by the client, submitted, and then entered into the system. QETE can report on the volume of tasks per year (typically 300-400) and can also provide detailed reports on all the projects over time including safety and operational issues, which have a domino effect on other projects. QETE is therefore in a position to provide accurate historical reporting and conduct issues analysis to establish patterns and improve performance.

Provision of Relevant, Credible, and Timely Advice

A second indicator utilized to assess the effectiveness of the provision of advice was a qualitative assessment of whether the information provided was relevant, credible, and timely. As TAs, EMTs provide advice on all aspects of equipment management throughout the life cycle of the fleets/platforms.²⁶ The range of advice provided by EMTs includes, but is not limited to advice regarding the following:

• technical material;
• operational planning and field support;
• specialty engineering (e.g. configuration management);
• training/managing technical staff;
• costing and budgeting for operations; and
• policies, procedures, and technical instructions.²⁷

Due to the lack of performance indicators available for this activity, only technical and specialized advice provided to operators/maintainers on fleet related issues was evaluated. The evaluation team conducted interviews at the director level and above within DGLEPM and within the maintenance chain of command of the CA leadership to determine stakeholder perceptions. Overall, stakeholders indicated that the advice sought and provided was of significant value. The program was further seen to provide sound technical advice and critical technical information that was not otherwise readily available to stakeholders.

Table 4. Business Planning Tools Used by DGLEPM. This table demonstrates the numerous tools currently used within DGLEPM for program planning and reporting across three main resource areas.

Key Finding 7: A patchwork of standalone tools limits the ability of DRMIS and HRMS to support decision makers.

A number of these tools duplicate functionality that is supposed to be integrated in DRMIS, namely, the Business Forecasting Tool, the Distribution Resource Planning Tool, and the fleet specific fleet management tools. In the case of human resources (HR) data, Peoplesort is a standalone database developed for HR planning purposes as a result of perceived issues with HRMS.

The Department does possess a data management system that would be able to provide such support. The Department has initiated an Enterprise Resource Planning system known as DRMIS, which is designed to provide the “integration of financial, material, workforce, and operations data thereby enhancing the DND/CAF’s capability in making strategic, operational, and tactical decisions founded on real time information.”²⁸ Accordingly, DRMIS should be a single reliable data source for business information, which would ensure that all decision makers act on the same information.

The evaluation noted, however, that DRMIS has not been utilized to any large extent to manage the various fleets. Interviewees stated concerns regarding data integrity of DRMIS (input errors, inaccurate disposal and spare parts records, and lack of reliability and failure information) when compared with existing tools. As a work around, several users exported the data from DRMIS and corrected any data errors. Although this addresses the existing business need of the user, it does not rectify the data errors in DRMIS. The proliferation of standalone tools and separate databases also undermines the ability of DRMIS to be used according to its mandate.

This has been a long-standing issue. In 2011, the OAG report on “Maintaining and Repairing Military Equipment – National Defence” noted the lack of reliable performance information as a significant concern. The report noted the following:

The OAG report recommended that DND develop the ability to produce overall and fleet-specific total cost information for its repair and maintenance activities, and that it implement the capacity to provide information on the performance and impacts of maintenance and repair activities.³⁰ DND agreed with the two recommendations pertaining to the “allocation and monitoring of financial resources” and responded that the requirements would be met by the implementation of DRMIS. To date, effort and progress have been made on the implementation of DRMIS; however, there are still several challenges that need to be addressed to integrate the use of DRMIS as an effective management tool into materiel management processes.

2.4.2 Immediate Outcome 2: Ability to meet the priorities and requirements of the CA and common equipment users

A key outcome of the LEP is to meet the sustainment priorities and support the readiness requirements of the CA. To assess this outcome, the following performance measures were utilized:

• the extent to which DGLEPM is aligned with the CA at the strategic level;
• the extent to which CA readiness requirements and priorities are met; and
• the extent to which DGLEPM governance can identify and address potential issues.

DGLEPM/CA Strategic Alignment

Key Finding 8: The LEP was able to support the strategic level priorities and requirements of the CA.

The extent to which the priorities of the LEP are aligned with the high-level priorities of the CA is a key measure of the effectiveness of the program. To ensure strategic alignment, DGLEPM works in close collaboration with the CA and its associated strategic requirements boards. Direction on readiness and priorities are communicated to DGLEPM through various strategic documents.³¹

To assess the extent to which the program and CA priorities were aligned, the evaluation utilized information from stakeholder interviews, focus groups, operational documentation, and lessons learned. Interviewees stated that over the past five years, the LEP was highly effective in supporting the CA in achieving its strategic objectives. During the evaluation period, these objectives principally revolved around sustaining the CA units that were deployed in Afghanistan.

Over the timeframe examined by the evaluation, the LEP successfully supported the sustainment of 11 large scale rotations in Afghanistan. Thousands of personnel and multiple vehicle fleets were deployed, maintained, and operated thousands of kilometres overseas and in extremely harsh conditions. Vehicle fleets were upgraded and customized in theatre to meet local conditions and needs, and new fleets were rapidly brought into service.

Going forward, DGLEPM continues to work closely with its CAF counterparts to optimize their resource allocation in light of projected readiness and operational needs, especially with the reality of a shrinking NP funding envelope.³²

CA Ability to Meet Stakeholder Requirements

Key Finding 9: While the requirements of the CA have been met in support of operations, the ongoing communication of direction, priorities, and requirements to DGLEPM is limited.

Equipment is an essential component of CA readiness.³³ As such, a key indicator of the effectiveness of the LEP is the ability to consistently meet the needs of its principal stakeholder, the CA, with respect to its readiness requirements. This includes sustaining the Army’s materiel requirements, including vehicle fleets and supporting equipment, for both training and operational deployments.

The CA rotates its level of readiness throughout its formations. Essentially, one component of the Army will be on “high readiness,” meaning fully equipped, trained, and available for operational deployments. Another component will be on a secondary level (“Road to Readiness”), meaning that it will be training up through various categories towards high readiness. Equipment numbers and condition are generally not at as high a standard as those for high readiness units. Lastly, a third component of the Army will be in a “reconstitution phase” pending recovery from high readiness or a deployment and only hold modest numbers of available equipment.

The CA monitors and tests its high readiness units on a frequent basis and, where issues arise, quickly moves to resolve them. With respect to the equipment component of high readiness units, the evaluation found through an assessment of documentation and stakeholder interviews that the LEP is meeting the CA’s needs for high readiness. This has further been demonstrated repeatedly by the ability of the CA to successfully deploy on numerous operations throughout the evaluation period.

Aside from high readiness requirements for operational deployments, the evaluation noted an apparent communication gap with respect to the articulation of equipment priorities and requirements. Interviewees indicated that the CA does not adequately communicate its priorities for equipment managed within DGLEPM beyond the general fidelity provided by strategic level documents. As such, there are no formal requirements for fleet or equipment availability, except for those that would be immediately needed for deployment. Formal sustainment requirements for normal training are virtually non-existent. Essentially, CA sustainment priorities are reactive and deal with equipment needs and issues as they arise.

Further, interviewees indicated that it is a challenge to provide adequate numbers of vehicles and equipment to be made available for the field force and the training system to deliver and sustain all training and operational commitments. Where assets are insufficient to satisfy all needs, operational commitments and the associated training to prepare for them take precedence over lower-priority training.

A fleet management strategy provides the linkage from CA operational readiness requirements to ADM(Mat) sustainment priorities. Equipment readiness is one of the pillars of operational readiness. Clearly defined equipment readiness requirements from the CA can provide the required guidance and direction to the DGLEPM fleet managers to generate and align their respective fleet management strategies in order to ensure resources are utilized in accordance with CA priorities.

Without clear direction and sustainment priorities, DGLEPM directorates base the requirements for the various fleets on historical data and usage forecasts.³⁴ The impact of such a practice is that there is no true ability to effectively manage resources to support fleets based upon priority. While the program has met the operational demand of the CA, without a clear overall fleet management strategy, it is difficult to determine optimum fleet sizes, states of readiness, and lifecycle management.

DGLEPM Governance and Issue Management

Key Finding 10: DGLEPM has an effective management plan to respond to and track major issues. However, there are challenges in the proactive oversight and management of routine equipment issues.

The ability to adapt to changes and address issues in an agile manner is another performance measure of the program’s ability to meet stakeholder needs. A responsive governance structure that can be proactive in order to mitigate challenges and address issues is important to the overall success of the program. Accordingly, the evaluation reviewed the management structure and approach of the LEP.

During the period of the evaluation, DGLEPM was operating within a challenging environment as the program sought to meet the demands of the Afghanistan mission, which included increased pressures in terms of volume and technical expertise.³⁵ This situation required a very flexible sustainment structure to adapt to ongoing operations. Sustainment issues were typically resolved at the LCMM level where there were mechanisms in place to escalate the issues through the chain of command as required. Based on several interviews, the majority of issues escalated by the LCMMs were related to safety of personnel in the field or had high visibility/risk.

In addition, DGLEPM has recently been challenged to define a steady state structure that satisfied the spirit of senior DND management efficiency initiatives throughout the evaluation period.³⁶

DGLEPM uses meetings as the mechanism to provide oversight and direction for ongoing issues. Through the EMT coordination meeting, held every two weeks, DLEPS receives updates and tracks key issues. A review of the tracking tool from these meetings (the “ops tracker”)³⁷ shows that issues raised by LCMMs, the field environments and other commands relating to safety, high risk, or those with high visibility are tracked. There is little evidence of oversight and guidance on issues not specifically raised. This creates a reliance on identification of problems, and allows only for management to react to problems once they occur. DGLEPM requires proactive performance mechanisms to be included in the performance measurement strategy framework.

2.4.3 Immediate Outcome 3: Disposal of equipment and materials in compliance with legislation

Disposal is the last phase in the life cycle management of a piece of equipment. The decision to dispose is based on the determination that a given asset is “no longer the most cost-effective means to address the CF’s requirement”.³⁸ When the decision to dispose has been made, disposal activities are to be completed in a timely fashion to avoid additional costs with respect to the sustainment and warehousing of the equipment.

Disposal requirements will differ depending on the nature of the goods to be disposed. For example, goods subject to International Traffic in Arms Regulations must undergo a more stringent set of disposal procedures than unregulated goods. The complexity of the disposal process can have an impact on the cost, duration, and disposal options for a given piece of equipment.

To evaluate the effectiveness of disposal activities, the following indicators were used:

• the percentage of schedule slippage against target milestones in the disposal plans; and
• the perceptions/evidence of the adequacy of material disposal.

Key Finding 11: There is a large volume of equipment and associated items waiting to be processed for disposal.

The DGLEPM Disposal Plan tracker indicated that 80 major line items were scheduled for disposal. This represents more than 42,000 single items. Table 5 shows that 30% of the projects planned have not commenced and that 26.3% of the projects have completed the disposal process.

The number of physical items to be disposed is significantly larger than what is currently tracked. The Disposal Plan does not account for the numerous minor items stored in depots and classified as repairable or dormant that are unlikely to be used and that should be disposed. The CA reports that there are 10 years of accumulated surplus materiel to be processed for disposal. ³⁹ The time and effort required to address this backlog is largely impeded by the lack of resources assigned to execute the Defence Renewal Team’s inventory rationalization recommendation.⁴⁰

Table 5. DGLEPM/CA Disposal Status Tracker. This table shows the status of the 80 major line items scheduled for disposal.

Interviews revealed that while disposal is generally acknowledged as a component of life cycle management, it is assigned a low priority. At the LCMM level, disposal is recognized as a responsibility but one that rarely gets acted on in a timely fashion because it is not directly related to readiness. As a result, there is very little incentive for LCMMs to prioritize disposal over other in-service support activities. Feedback from the LCMM questionnaire shows that 61% of LCMMs spend less than 10% of their time on disposal activities. LCMMs stated that their primary disposal challenges were as follows:

• competing higher priority work demands for in-service support; and
• time intensive nature of the disposal process.

Key Finding 12: The lack of performance indicators across the multiple DND/CAF organizations involved in the disposal process inhibits the ability to measure performance.

The disposal process has sufficient guidance documentation at multiple levels. DND promulgated a Defence Administrative Order and Directive (DAOD)⁴² on the subject. The CA updated a Materiel Disposal Policy in October 2013 to provide an overview and essential guidance on the CA equipment divestment process. ADM(Mat) also provides disposal guidance documentation. In addition, DGLEPM published a guidance document covering the disposal of major equipment to enhance the disposal process for the CA and common equipment.

The disposal process involves all the stakeholders listed in Section 1.1.4 of this report. Interviews and focus groups indicated that the complicated interdependencies of the disposal process and the transfer of responsibility impede the effective execution of disposal activities. As a result of this complex process, materiel is kept within the system longer than it should be, unnecessarily increasing the life cycle costs.

Interviewees proposed that centralizing the DGLEPM divestment function into a specific business unit would allow for a better focus and would amalgamate the different phases of the disposal process. The 2012 internal LEP review proposed an initiative to investigate this option.

2.4.4 Immediate Outcome 4: Ensuring land and common equipment is safely stored and operated

Most safety-related regulations, standard operating procedures, and repair instructions are developed during the equipment acquisition phase, prior to entry into service.⁴³ The management and oversight of in-service equipment safety is the decentralized responsibility of users at the unit level. At this level, safety is monitored by annual inspections that certify the safety of the equipment by local technicians at first and second line maintenance, which belong to the operational commands. Safety monitoring also consists of daily and monthly inspections by the operators. DGLEPM liaises with the CA headquarters staff to establish safety doctrine and manage feedback on the fleets.

For in-service equipment, DGLEPM has little control over equipment safety; however, it is responsible for the CA safety policies within DGLEPM.There are two broad domains of materiel safety management within DGLEPM. These are equipment safety and ammunition safety. Due to the nature of ammunition, safety issues are dealt with under a separate policy and procedural framework.

The indicators used to assess this outcome are as follows:

• evidence of oversight and management to contribute to materially safe equipment; and
• evidence of oversight and management to contribute to materially safe ammunition.

Materially Safe Equipment

Key Finding 13: There do not appear to be clear roles and responsibilities for equipment safety in DGLEPM.

From a DGLEPM perspective, safety issues are brought to the attention of senior management through the technical failure and unsatisfactory condition report processes for user-reported problems. DGLEPM responds to equipment safety issues in a reactive manner and becomes involved through the escalation process. Apart from shared working groups, such as the international Light Armoured Vehicle Working Group, there is no proactive surveillance of equipment safety by DGLEPM.

Based on the information available at the time of this evaluation, it is unclear whether the LEP would benefit from having greater visibility into equipment safety through proactive safety monitoring. To this end, there is need for greater clarity concerning the roles and responsibilities for equipment safety. DGLEPM staff indicated that greater visibility into fleet safety issues is desirable but there is no well-defined framework or governance structure to facilitate this.

One notable difference between the LEP and equipment programs for the RCAF is the role of QETE. QETE has established policies and service standards with the RCAF that provide a mandate to investigate any major vehicle related incident. In the case of the CA, there is no policy or service standard for QETE to investigate accidents. QETE is brought in only when requested. Interviews indicate that formalizing the role of QETE in accident and incident response would be beneficial and provide a more timely response to investigations.

Materially Safe Ammunition

Key Finding 14: DGLEPM has clear roles and responsibilities to execute an effective program to address ammunition safety requirements.

DGLEPM manages a multi-level ammunition safety organization. DAEME has a role in managing ammunition and explosive safety and suitability requirements. DAEME is also responsible for establishing standards for ammunition and explosive design and life-cycle requirements, safety and suitability for service assessment requirements, as well as Qualified Ammunition Technical Authority training relating to ammunition and explosive engineering and technical requirements.⁴⁴ The Ammunition Safety and Suitability Board, chaired by DAEME, provides impartial validation of the safety and suitability for service of ammunition and explosives.⁴⁵

In 2012, the Ammunition Program Restructure initiative sponsored by the Materiel Group commenced to establish a new program to provide a more effective and efficient way to manage DND/CAF ammunition and explosives. The initiative is aligned with the Defence Renewal effort in that it will improve business processes and activities related to ammunition, resulting in maximized operational capability and readiness.

2.5 Performance—Demonstration of Efficiency and Economy

The findings in the following section are based on an assessment of the efficiency and economy of the program—or an assessment of whether existing resources (i.e., financial and human)—are optimally utilised to achieve the required results.

As stated in previous sections, within DGLEPM there is a lack of a tracking data and a minimal use of the enterprise data management systems. While costs per fleet can be identified, they are not tied to any outputs. For example, while it is possible to determine how much is spent each year for the Leopard Tank fleet (or at least the DGLEPM portion), there is no data that can be attributed to it, such as hours of operation, state of readiness, and vehicle availability. For fleets with larger numbers of vehicles, the data is even more unreliable or scarce.

Accordingly, it is extremely difficult to determine whether the program has become more or less efficient or whether appropriate funding is even being expended. As such, to evaluate the efficiency and economy of the program, the evaluation examined how key resources were managed. The indicators utilized included the following:

• the extent to which DGLEPM has implemented a Performance Measurement Framework;
• the extent to which DGLEPM manages human resources; and

• the extent to which DGLEPM manages materiel resources.

2.5.1 Performance Measurement Strategy Framework

Key Finding 15: DGLEPM lacks a Performance Measurement Strategy Framework. This limits the ability of the organization to identify and implement economies and efficiencies based on sound business intelligence.

One of the principal indications of efficient program management is the existence of a sound performance measurement framework than can tie resources to outputs and outcomes and the utilization of such a framework for management decisions.

The evaluation could find no evidence of a DGLEPM Performance Measurement Strategy Framework. While some data analysis and performance management activity may be occurring at the LCMM level, there is at present no structure to ensure the right data is collected and acted upon to facilitate the improved economy and efficiency of the program. Data management systems such as DRMIS exist; however, within the DGLEPM group challenges exist concerning their utilization, including training, data availability, and analytical skills. In addition, due to the numerous stakeholders involved in equipment maintenance, any performance measurement activity will be dependent upon multiple external players (CA, finance, suppliers), and a horizontal approach will be required.

Without such a system, the program is not able to adequately equate resources and effort to activities, thus compromising the ability to optimize resource utilization and to understand the true costs associated with individual fleets, equipment, disposal, and various standards of readiness.

2.5.2 Human Resources Management

Key Finding 16: The lack of sound data on the numbers of contracted personnel creates significant challenges in program productivity management.

Key Finding 17: Staff attrition is creating significant knowledge transfer issues.

The effective utilization of personnel is a key indicator for the performance of the LEP. As noted in Section 1.2.3, within DGLEPM there are approximately 952 civilian and 273 military employees. In addition, there are numerous contracted personnel who work in sustainment related activities. From FY 2008/09 to FY 2012/13 the total number of civilian and military personnel increased by 7.2%. The number of contractors is not tracked. However, Table 2 shows that professional services funding has also increased from $3 million to $16 million during this period. It is suspected that some of these expenditures were for fleet or material service activities (i.e., the conduct of repairs, engineering support, or new capital design and specification work), the overall value of which is quite large. This indicates that professional services contractors play a significant role in maintenance management activities, likely in roles that DND/CAF employees used to perform.

The lack of availability of actual worker numbers is problematic from a management perspective. Assessing the productivity of employees without this key data is not possible. For example, the total amount of expenditures in the program has declined by 12% over the study period, or even as much as 22% after factoring in general inflation. Yet the numbers of military and civilian employees has increased by 7.2%, indicating a significant reduction (27% factoring inflation) in overall productivity over a five-year period. The increase in DND/CAF employees may be offset by reduced numbers of contractors, but there is no data that supports that assumption and no evidence that there has been a reduction in professional services contract expenditures over this period (although they have declined by 10% from 2013 onwards).

During the last year of the evaluation period, the program implemented work-force adjustments through the Deficit Reduction Action Plan initiatives. However, the increasing demands for in-service fleet management, alongside the pressure to reconstitute the fleets in a post-Afghanistan context, is creating HR challenges. ADM(Mat) implemented the Deficit Reduction Action Plan initiatives through natural attrition (retirements) rather than through work force adjustment.

Multiple interviews with program staff indicated that the attrition approach created problems. Considerable knowledge, skills, and experience are required to effectively manage the various fleets within the LEP. Some of this knowledge is technical in nature, but much of it is corporate knowledge about the history of the fleets. In addition, there are several positions that require specialist training. However, the inability to hire replacements and the lack of DGLEPM-enforced personnel transfers impeded the ability to sustain certain capabilities. The consequence is that many experienced staff members have left the organization without passing on their expertise or knowledge. DGLEPM relies heavily on experienced senior staff to mentor new employees. The loss of experienced staff due to the Deficit Reduction Action Plan, in combination with an increasing workload, means there is less time to support critical informal training.

Compounding this risk is the fact that there are currently 121 people eligible to retire without penalty, and this number will grow to 370 over the next five years. Approximately 44% of LCMMs and SMs have less than three years of experience.⁴⁶

Efficient succession planning is also required for the military members of DGLEPM. Military members enter into their roles with a rich understanding of the military. Nevertheless, respondents said that it is a challenge for them to learn the new business processes and the organizational approaches to manage the equipment. By and large, due to course availability, it could take a new military member up to two years to become fully trained. As most military staff will rotate to a different position or organization after three years, there is only one year of full employment efficiency.

Further, in the past, Director of Land Requirements (DLR) was the CA link for both the acquisition of new equipment and in-service support. Over the past few years, as a result of personnel reductions, DLR has shifted its focus to the procurement of new equipment. Positions were transferred from DLR to CA headquarters staff with the expectation that they would assume the in-service support responsibilities. Interviews indicated that CA headquarters staff have not assumed all of the in-service support responsibilities previously held by DLR.

2.5.3 Materiel Resources Management

In the supply system, there are multiple controls in place that allow effective oversight of life cycle management. To assess the efficiency of the material resources management within LEP, three areas were examined by using purchase order data provided by Director Supply Chain Operations/Distribution Resource Planning (DRP) Team, as follows:

1. ordering⁴⁷ materiel when the same materiel is waiting to be repaired in various DND locations or sitting in depots/warehouses pending repair (ordering for repairable);
2. ordering materiel when the result of the procurement will exceed the maximum normally established for that item (ordering in-excess); and
3. ordering items when those items exist in a “dormant stock” category (ordering dormant).

Key Finding 18: There are significant occurrences of purchase order activities for items that are already in stock but needing repair, and that are above the maximum identified allocation or that have not been used for at least four years.

Generation of Purchase Orders for Repairable Items

According to DRMIS data, there is a large quantity of items in the inventory classified as repair reserve. Some of these items are obsolete and require disposal. Evidence shows that there is a significant amount of monthly purchase orders processed to buy materiel that is presently categorized as repair reserve. LCMMs have decided not to repair these parts but to hold onto them and to put them back into service.

Figure 1 illustrates the total amount of procurement orders placed for repairable items during the period of October 2010 to July 2014. The average monthly purchase amount for the 46-month period is $6,962,080.98.

Figure 1. Monthly Purchase Orders Processed to Buy Repairable Items.⁴⁸ This figure illustrates the total amount of procurement orders placed for repairable items during the period of October 2010 to July 2014.

Although it is recognized that there are many factors involved in deciding whether to repair items, dispose of them, or replace them with new items, there is no evidence that this type of data is being used to provide appropriate direction to improve effectiveness and efficiency.

Ordering in Excess

Supply management allocates specific minimum and maximum levels for all items to be kept in inventory. The minimum and maximum levels ensure that an appropriate level of stock is maintained and available at all times.

Figure 2 identifies, for the period from October 2010 to July 2014, the amount applicable to procurement orders placed for quantities of materiel that will result in an on-hand balance that will exceed the determined maximum inventory holding levels. This represents purchase orders created in a previous month for items catalogued for at least four years and having a projected system-wide ending inventory quantity above the recommended maximum holdings for the order due month. The average monthly amount applicable to the purchase orders placed for the period identified is $10,389,681.90.

Figure 2. Monthly Purchase Orders Processed to Buy in Excess.⁴⁹ This figure illustrates procurement orders placed for quantities of materiel that will result in an on-hand balance that exceeds the determined maximum inventory holding levels.

Ordering Dormant Stock

Figure 3 below identifies procurement orders placed for items that are considered dormant. Dormant items are those that have not been requisitioned for the past four years by any CAF customer accounts and that do not have any projected demand for the next six years. The average monthly amount of purchase orders for dormant stock for the 46-month period is $415,320.

Figure 3. Monthly Purchase Orders Processed to Buy Dormant Stock.⁵⁰ This figure illustrates procurement orders placed for items that are considered dormant.

For the three areas of repairable items, in-excess, and dormant stock, the total amount of processed purchase orders results in a yearly average of $213 million. This represents almost 20% of the average annual expenditures of the whole LEP. Although the trend analysis demonstrates a decline in the amount of purchase orders, and there are several possible reasons for the occurrences, the analysis underlines the fact that there is still work to be done for more efficient practices to be enforced in the management of the LEP.

Annex A—Management Action Plan

Key Finding 5: The lack of availability and reliability of data limits DGLEPM’s ability to effectively and efficiently track and manage the provision of advice.

Management Action

The provision of advice and guidance is a core activity of the Army and common equipment sustainment program and is provided proactively based on the ongoing assessment of equipment health or reactively upon demand. As stated in this evaluation report, the provision of advice sought and provided was of significant value. The effort associated with the development, implementation, and reporting, of a comprehensive Performance Measurement Strategy Framework is significant and will take time to fully implement. DGLEPM’s initial focus will be the establishment and monitoring of performance indicators related to operational readiness and equipment safety. Indicators for these measures will be in place by April 2017. DGLEPM is monitoring an initiative within the maritime equipment program to track and assess workload distribution. The outcomes and lessons learned from this initiative will also assist in the development of advice-related performance indicators.

OPI: ADM(MAT)/DGLEPM

Target Date: Performance Measurement Strategy Framework to be completed by April 30, 2016. The establishment and reporting of performance indicators, including the provision of advice, to be completed by April 30, 2017.

Key Finding 7: A patchwork of standalone tools limits the ability of DRMIS and HRMS to support decision makers.

Management Action

In support of program level decision making, DGLEPM is committed to improving the use and data integrity of DRMIS. Specifically, the Performance Measurement Strategy Framework, once established, will leverage available DRMIS reporting capabilities. DGLEPM also has commenced a multi-year plan to review master LEPM DRMIS records and correct data anomalies.

OPI: ADM(Mat)/DGLEPM

Target Date: Master record review to be completed by March 31, 2017. DGLEPM Performance Measurement Strategy Framework to be completed by April 30, 2017.

Management Action

In support of Defence Renewal 3.2 (Application Portfolio Management), DGLEPM is engaged in a multi-year review process to reduce and consolidate material acquisition and support applications used in support of program activities and to maximize the use of DRMIS and HRMS enterprise systems. DGLEPM is engaged in articulating system requirements to more effectively manage human resources and support program-level decision making but these enterprise-level modifications are dependent upon corporate resourcing and prioritization.

OPI: ADM(Mat)/DGLEPM

OCI: Assistant Deputy Minister (Information Management)

Target Date: DGLEPM application rationalization to be completed by 31 March 2018.

Key Finding 9: While the requirements of the CA have been met in support of operations, the ongoing communication of direction, priorities, and requirements to DGLEPM is limited.

Management Action

Decision making at the program level pertaining to the acquisition, maintenance, and disposal of equipment is complex and involves multiple stakeholders including, but not limited to, Chief of Force Development, the Environmental chiefs, and ADM(Mat). In this context, ADM(Mat) is not responsible for determining when an equipment fleet will be replaced or disposed of, but instead is responsible for sustaining the equipment during its life cycle. The equipment strategy concept in DGLEPM was initiated to ensure that decision making in support of fleet sustainment is logical and aligns with Army equipment requirement priorities. The first fleet strategy pilot was launched in FY 2013/14. In FY 2014/15, seven equipment strategies were completed and briefed to the CA for its endorsement. In FY 2015/16, this initiative is being expanded to complete 10 additional equipment strategies. By March 2018, a total of 30 equipment strategies, covering the LEP will be implemented. In addition, DGLEPM continues to improve its engagement with the Canadian Army and institutionalize this process through planned briefs at the Army Capability Development Board in summer 2015 and via the release of an implementation instruction by March 2016. Once a steady state is achieved, equipment strategies will be updated every three years.

OPI: ADM(Mat)/DGLEPM

OCI: Commander CA

Target Date: Equipment Strategy Implementation Instruction to be released by March 31, 2016. Equipment strategy implementation to be completed by March 31, 2018.

Key Finding 10: DGLEPM has an effective management plan to respond to and track major issues. However, there are challenges in the proactive oversight and management of routine equipment issues.

Management Action

DGLEPM has bi-weekly EMT coordination meetings to identify and ensure resolution of equipment issues and high priority initiatives. Meeting activities are recorded and monitored in a tracking document. The implementation of equipment management strategies and the establishment of a Performance Measurement Strategy Framework, including the establishment of material management performance measures, will enable the proactive management of equipment fleets. Technical system integration and issue resolution is also occurring at the program level via Equipment Program Boards, including for command, control, communications, intelligence, surveillance, and reconnaissance. DGLEPM is considering applying this model to other platforms.

OPI: ADM(Mat)/DGLEPM

Target Date: Establishment of material management performance indicators in support of the DGLEPM Performance Measurement Strategy Framework to be completed by April 30, 2017.

Equipment strategy implementation to be completed by March 31, 2018.

Key Finding 11: There is a large volume of equipment and associated items waiting to be processed for disposal.

Management Action

During the Afghanistan mission, DGLEPM’s priority and effort was focussed on the acquisition and sustainment of capabilities in support of the operation. Since the termination of operations in Afghanistan, DGLEPM has refocused efforts on other program activities such as the disposal backlog. Specifically, DGLEPM has centralized disposal coordination and monitoring and has updated its disposal plan, which includes disposal target timelines. The division currently participates in Land Equipment Disposal Working Groups to coordinate and maintain momentum on disposal activities. Progress is monitored monthly.

OPI: ADM(Mat)/DGLEPM

Target Date: Inclusion of disposal milestones in the DGLEPM Performance Measurement Strategy Framework to be completed by April 30, 2017.

Key Finding 12: The lack of performance indicators across the multiple DND/CAF organizations involved in the disposal process inhibits the ability to measure performance.

Management Action

From 2010 to 2012, ADM(Mat) processed, mapped, and modernized the departmental disposal program. Through this process, the team involved in that activity realized that the effectiveness of the departmental program could not be optimized until dependencies on related inventory and materiel management processes were also modernized. This led ADM(Mat) to launch the Inventory Management Modernization and Rationalization Project, which is a five-year project aimed at rationalizing dormant stock and modernizing inventory management across the Department from end to end. One of the deliverables of this project is to develop KPIs for inventory management processes that include disposal functions. A study was recently conducted by the Project team on disposal processes across the DND/CAF. This particular sub-project of the Inventory Management Modernization and Rationalization Project was concluded on March 31, 2015. It aimed to both further modernize disposal processes to increase efficiency and effectiveness by identifying further gaps and to recommend courses of action to reduce currently held stock identified for disposal. The modernization study reviewed disposal from a programmatic approach, including planning and reporting, program enablers, service delivery, compliance, risk management, decision support, knowledge management, and policy and process. Process mapping of the modernized process will be conducted in May and June 2015.

OPI: ADM(Mat)/DGMSSC

Target Date: Completed March 31, 2015.

Key Finding 13: There do not appear to be clear roles and responsibilities for equipment safety in DGLEPM.

Management Action

In October 2014, DGLEPM launched a strategic initiative to investigate the implementation of a Land Materiel Assurance program in order to ensure CAF equipment is materially suitable and safe. In addition, the program would formalize the provision of advice to operational commanders on technical issues affecting equipment availability and enable material assurance reporting. Although this strategic initiative is still ongoing, early indications are that the Land Materiel Assurance program will benefit the overall LEP and provide an effective and sustainable framework to integrate existing processes and governance while ensuring appropriate assurances throughout the equipment life cycle. Although proactive materiel assurance activities are currently performed at the tactical and strategic levels, it has been noted that formal governance is lacking and will be implemented as part of the Land Materiel Assurance program. This governance will define accountabilities, responsibilities, and authorities for both proactive and reactive sustainment activities, leverage existing equipment inspections (annual technical inspections and Life Cycle Materiel Manager visits) and will be synchronized with equipment strategies.

OPI: ADM(Mat)/DGLEPM

Target Date: Land Materiel Assurance program development to be completed by August 31, 2016. Program launch to be completed by August 31, 2017.

Key Finding 15: DGLEPM lacks a Performance Measurement Strategy Framework. This limits the ability of the organization to identify and implement economies and efficiencies based on sound business intelligence.

Management Action

DGLEPM has initiated efforts to establish a program coordination function to increase cross-coordination and has initiated program-level reporting to support decision making pertaining to finance and human resources. DGLEPM, with the assistance of ADM(RS), will establish a Performance Measurement Strategy Framework to support program-level decision making and identify potential economies and efficiencies. This framework will leverage DRMIS and HRMS reporting capabilities.

OPI: ADM(Mat)/DGLEPM

OCI: ADM(RS)

Target Date: DGLEPM Performance Measurement Strategy Framework to be completed by April 30, 2016.

Key Finding 16: The lack of sound data on the numbers of contracted personnel creates significant challenges in program productivity management.

Management Action

DGLEPM will investigate total level of effort (civilian, military, and embedded contractor support) data, as well as internal and external factors that influence productivity, to determine if there has been a decline in overall output.

OPI: ADM(Mat)/DGLEPM

Target Date: Output investigation to be completed by April 30, 2017.

Key Finding 17: Staff attrition is creating significant knowledge transfer issues.

Management Action

To ensure day-to-day business continuity and corporate knowledge transfer, DGLEPM has implemented a divisional directive to ensure a comprehensive reporting from out-going to incoming directors is completed. In addition, a formal position handover process is under consideration at the group level. DGLEPM will also implement a system to track personnel training requirements by identifying essential knowledge, skills, and training for key positions. This system will permit the identification of performance indicators such as the percentage of employees holding requisite experience and having completed essential training for their respective positions. This will permit DGLEPM to monitor its ability to continue business without undue risk due to knowledge loss and will be an essential component in succession planning by assuring us that the organization is preparing the right people with the right knowledge for their next position.

OPI: ADM(Mat)/DGLEPM

Target Date: Establishment of knowledge and training performance indicators into the Performance Measurement Strategy Framework to be completed by April 30, 2017. Knowledge Management tracking implementation to be completed by April 30, 2017.

Key Finding 18: There are significant occurrences of purchase order activities for items that are already in stock but needing repair, and that are above the maximum identified allocation or that have not been used for at least four years.

Management Action

Monthly monitoring of purchase activities is ongoing. All indicators are currently trending downward. Any anomalies identified are investigated with the EMT, and data integrity issues are addressed. To date, investigations have determined that the high majority of purchases are legitimate activities in support of equipment capabilities and the LEP. Such examples, include, but are not limited to, parts to address obsolescence management, legitimate life time buys, and increased requirements associated with a forecasted operational surge. Performance indicators for purchase activities will be included in the Performance Measurement Strategy Framework that is developed in conjunction with ADM(RS).

OPI: ADM(Mat)/DGLEPM

Target Date: Establishment of material management performance indicators in support of the Performance Measurement Strategy Framework by April 30, 2017.

Annex B—Evaluation Methodology and Limitations

Methodology

Overview of Data Collection Methods

The evaluation findings and associated recommendations are supported by multiple lines of evidence collected through qualitative and quantitative research methods. Information and data collected from relevant sources were analyzed to inform conclusions on the relevance and performance (effectiveness and efficiency/economy) of the program.

Key activities of the LEP were grouped together to determine the expected outcomes of the program as defined in the logic model. KPIs were identified for each outcome to assess LEP performance. Outcomes and their associated KPIs are listed in the Evaluation Matrix, located at Annex D.

The data for each KPI was collected using the following research methods:

• Literature review
• Document review
• Key informant interviews
• Site visits
• Administrative and financial data reviews
• Benchmarking
• Questionnaire
• Focus groups

Details on Data Collection Methods

Literature Review

A literature review was undertaken to identify the existence of similar programs in other government departments, in industry, or in other countries. The aim was to provide context, identify methods and approaches used to evaluate similar programs, and identify known best practices and expected results for benchmarking.

Document Review

A document review was completed to provide a complete picture of the purpose, scope, and mandate of the LEP. Additionally, the document review enabled visibility into the larger organizational and operational context in which the program operates (DND/CAF). Core program documents were used to establish parameters regarding program delivery and as a source of data to support the performance evaluation of the economy and efficiency of the program.

The following documents were reviewed during the conduct phase of the evaluation:

• policy, legislation, and related accountability documents: the TBS Policy on Evaluation and the Directive on the Evaluation Function, DAODs, Departmental Performance Reports, Reports on Plans and Priorities, the Defence Procurement Strategy, and the Procurement Administration Manual;
• strategic documents, reviews, and plans: DGLEPM Business Plans, Strategic Plans, Campaign Plans, Force Posture and Readiness, LEP Review, Defence Renewal Charter;
• GC direction and related documents: CFDS, Speeches from the Throne;
• program specific documents: minutes from senior level meetings, reporting documents, handover briefings, training manuals, extractions from the corporate website, and related knowledge transfer materials;
• other government/military documents: reports, papers, and related documentation that enabled comparisons between the LEP and similar organizations in allied militaries; and
• other audits and evaluations.

Key Informant Interviews

Most of the stakeholders that were interviewed were located within the National Capital Region, limiting the transportation requirements and facilitating the alignment of agendas. Nineteen interviews were conducted with key senior stakeholders from the LEP (director general and directors) and key stakeholder groups, including PWGSC and the CA. There were a total of 35 interviewees. Table B-1 provides the details.

Table B-1. Number of Interviews and Interviewees by Organization. This table summarizes the number of interviews conducted and the stakeholders interviewed.

The interviewees were given an interview guide in advance. Interview guides were tailored to each group and interview questions were aligned with the evaluation matrix. All the interviews were conducted in person with the exception of the one with 7 Canadian Forces Supply Depot (CFSD), which was done by phone. The evaluators transcribed the notes taken during the interviews to create interview summary notes that were used to fill the evaluation matrix with relevant observations.

Site Visits

DGLEPM’s head office was visited regularly as most of the directorates are located in the same building, with the exception of QETE and 202 Wksp Dep.

The QETE on-site visit was very informative on the various methods and techniques used to achieve results.

202 Wksp Dep is a large facility located at Area Support Unit Montreal (Longue-Pointe), which performs repairs and modifications to Canadian military equipment used mainly by the CA. It is the only centre for third and fourth line overhaul of Canadian Land Forces equipment. It inspects, repairs, and maintains a wide array of equipment, from small electronic parts up to the overhaul of complex and sophisticated weapon systems, such as Leopard tanks. The onsite visit to 202 Wksp Dep enabled the team to observe all aspects of the full production capacity and capabilities of the third and fourth line repair workshop.

Some organizations not part of DGLEPM were also visited, namely the CA headquarters, PWGSC, and 25 CFSD.

The CA headquarters site visit allowed the evaluation team to assess the relevance and performance of the LEP from the main client’s point of view.

PWGSC is the procurement agency for defence equipment. The site visit to PWGSC allowed the evaluation team to assess the complementarities of activities between the two organizations and also to obtain data pertaining to general and contract expenditures.

The 25 CFSD is located at Area Support Unit Montreal (Longue-Pointe). It is a key component of the military supply chain and has the potential to influence the performance of the life cycle management of land equipment. It is responsible for receiving, validating, warehousing, ordering, picking, packaging, and shipping parts for most equipment, including the fleets within the control of DGLEPM. The Depot is the custodian of material, but not the owner/controller. The site visit to 25 CFSD allowed the evaluation team to see all elements of the CFSD warehousing and supply management process. In addition, the team was able to observe storage capacity related issues pertaining to items awaiting repair and overhaul or disposal.

Administrative and Financial Data Reviews

Financial and administrative data for the LEP were analyzed with respect to the defined indicators in the Evaluation Matrix.

The administrative data that was reviewed came from the following sources: extracts from HRMS Civilian (8.9) and HRMS Military (7.5), HR Planning Data (PWGSC Incumbent Files and HRMS extracts), Public Service Employee Survey, Management Accountability Framework, Public Service Management Dashboard, PeopleSoft, and various DGLEPM reports and planning documents (e.g., Business Plans).

The financial data for the LEP were obtained from: DRMIS, PWGSC Goods and Services Identification Number system, and DRP.

Benchmarking

At the outset of this evaluation, efforts were made to identify organizations suitable for benchmarking the LEP. The evaluation team made contact with equivalent programs within allied forces, namely, Australia, the United States, and the United Kingdom. The team was only successful in obtaining allied information from the Australian Army.

The evaluation also looked at relevant organizations with the responsibility of managing a major fleet of varying vehicles. Some insights were garnered from the Royal Canadian Mounted Police and from Canada Post, who manage large and relatively diverse fleets of equipment. In reality, the team learned that outside of military organizations, the size, vehicle types, and common usage scenarios are too different to allow for useful benchmarking insights on the LEP. Even within military organizations, there is a great deal of variance even with common vehicle types (e.g., light armoured vehicles) depending on the configuration and usage scenarios. Consequently, the team was unable to obtain sufficient benchmarking data to warrant a thorough benchmarking analysis.

Questionnaire

A questionnaire was administered to LCMMs, SMs, and TAs involved in the LEP. There are more than 200 employees occupying these roles in DGLEPM. The questionnaire format was chosen to enable an efficient means to survey the views of a large cross-section of the LCMM/SM/TA population.

Respondents are critical members of the EMTs responsible for the life cycle of the equipment and for the maintenance and in-service support of the fleets. Given their positions, they have a great deal of insight on the overall effectiveness of fleet management for the LEP.

The questionnaire was based on the Evaluation Matrix and questions were mapped to the corresponding indicators. The questionnaire topics covered were time spent on core activities, degree of client focus, inventory management, management tools, training, governance and corporate support, and stakeholder interaction.

More than 236 candidates were provided the questionnaire. There were 137 respondents included in the results after data cleansing and representing all the directorates involved in the LEP. The completed questionnaires were sent back to the evaluators by secure email, and then compiled manually in an Excel spreadsheet for a statistically designed analysis.

Focus Groups

Two focus group sessions of two hours each were hosted to accommodate a total of nine participants. These participants were from among the 137 LCMMs and SMs who had completed the questionnaire.

The aim of the focus groups was to obtain additional information to complement the results observed from the questionnaire administered to SMs, LCMMs, and TAs within DGLEPM. The focus group format allowed for a free conversation to delve more deeply into issues discovered in the questionnaire.

The focus group sessions were set up according to best practices. Participants were split into two smaller groups to maximize input from all participants. Standard rules of engagement and etiquette were communicated. Efforts were made to keep the discussions as open ended as possible to allow for the free flow of input from participants.

Limitations

This section identifies constraints and limitations of the evaluation (e.g., limitations of design, methods, and consultations, actual/potential biases, reliability of the data, and the impact on evaluation findings) and how the evaluation attempted to overcome them.

The limitations and mitigation strategies are summarized in Table B-2.

Table B-2. Evaluation Limitations and Mitigation Strategies. This table summarizes the constraints and limitations of the evaluation.

Annex C—Logic Model

Figure C-1. Logic Model for the LEP. This flowchart shows the relationship between the program’s main activities, outputs, and expected outcomes.

Annex D—Evaluation Matrix

Table D-1. Relevance – Continued Need. This table indicates the data collection methods used to assess the evaluation issues/questions for determining the relevance (continued need) of the LEP.

Table D-2. Relevance – Alignment with Federal Roles and Responsibilities. This table indicates the data collection methods used to assess the evaluation issues/questions for determining the relevance (alignment with federal roles and responsibilities) of the LEP.

Table D-3. Relevance – Alignment with Government Priorities. This table indicates the data collection methods used to assess the evaluation issues/questions for determining the relevance (alignment with government priorities and departmental strategic outcomes) of the LEP.

Table D-4. Performance – Effectiveness. This table indicates the data collection methods used to assess the evaluation issues/questions for determining the performance of the LEP in terms of achievement of outcomes (effectiveness).

Table D-5. Performance – Efficiency and Economy. This table indicates the data collection methods used to assess the evaluation issues/questions for determining the relevance and performance of the LEP.

___________________________________________________________________________________________________________________________

Footnote 1 Canadian Armed Forces website. http://www.forces.gc.ca/en/about-org-structure/assistant-deputy-minister-materiel.page. Last consulted February 20, 2015.

Footnote 2 The Equipment Management Team (EMT) Handbook.

Footnote 3 DND Performance Measurement Framework FY 2013/14.

Footnote 4 ADM(Mat) Level 1 Business Plan 2013-2014.

Footnote 5 DND Program Activity Architecture, April 2010. The DND Program Activity Architecture was updated and changed to the Program Alignment Architecture in April 2014.

Footnote 6 Ibid.

Footnote 7 CRS (former designation of ADM(RS)) conducted an Audit of Warehouse Management in 2013, and in the fall of 2011, the OAG Report included Chapter 5: Maintaining and Repairing Military Equipment – National Defence.

Footnote 8 In 2011, CRS published a report on the Evaluation of Land Force Readiness and Training. This report focused on the serviceability of the equipment, the proper qualifications for technicians, and whether or not technicians have the necessary materiel.

Footnote 9 Population is defined as the physical number of people whatever time schedule they are on, while FTE is defined as the number of full-time equivalent employee.

Footnote 10 Data extracted from a general ledger account.

Footnote 11 Extracts from HRMS – PeopleSoft.

Footnote 12 http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15681&section=text. Last consulted July 4, 2014.

Footnote 13 Department of National Defence and the Canadian Armed Forces Defence Renewal Charter, October 2013.

Footnote 14 NP briefing package for the Associate Deputy Minister, dated February 15, 2012.

Footnote 15 Defence Renewal Charter.

Footnote 16 Ibid.

Footnote 17 http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=12062&section=HTML. Last consulted on July 4, 2014.

Footnote 18 Defence Production Act, Part 1 – Defence Procurement. http://laws-lois.justice.gc.ca/eng/acts/d-1/. Last consulted on July 4, 2014.

Footnote 19 As stated in the Defence Production Act, PWGSC has the responsibility to acquire, utilize, store, transport, sell, exchange, or dispose of defence supplies, as well as to manufacture/produce, finish, assemble, process, develop, repair, maintain, or service defence supplies.

Footnote 20 Procurement Administration Manual, Revision 52, June 2013.

Footnote 21 CFDS. http://defenceteam-equipedeladefense.mil.ca/about-apropos/pri/cfds-sdcd/pdf/CFDS_English.pdf. Last consulted on July 4, 2014.

Footnote 22 Descriptors for GC Outcome Areas http://www.tbs-sct.gc.ca/ppg-cpr/descript-eng.aspx. Last consulted on July 4, 2014.

Footnote 23 ADM(Mat) Level 1 Business Plan 2013-14.

Footnote 24 Canadian Army Equipment Readiness Strategy, May 27, 2013.

Footnote 25 Data provided by DLEPS 4-5, April 2014.

Footnote 26 The EMT Handbook.

Footnote 27 Ibid.

Footnote 28 http://drmis-sigrd.mil.ca/aboutus-anotresu-eng.asp. Last consulted on July 4, 2014.

Footnote 29 2011 Fall Report of the Auditor General. Chapter 5: Maintaining and Repairing Military Equipment.

Footnote 30 Ibid.

Footnote 31 The Canadian Army Managed Readiness Plan, March 1, 2013, and the Canadian Army Equipment Readiness Strategy are strategic documents that link to force posture and readiness requirements.

Footnote 32 NP Process Review – Project Charter, January 2013.

Footnote 33 Canadian Army Equipment Readiness Strategy.

Footnote 34 Canadian Army Equipment Readiness Strategy.

Footnote 35 DGLEPM Business Plan 2009/10 and 2010/11.

Footnote 36 DGLEPM Business Plan 2009/10, 2010/11, 2011/12, and 2012/13.

Footnote 37 from SharePoint documents.

Footnote 38 EMT Handbook.

Footnote 39 Presentation at the Army Council, Kingston, August 2014.

Footnote 40 Defence Renewal Plan October 2013, Annex 2.1 – Inventory Management.

Footnote 41 Some of the projects in this category have not yet achieved their due date; it sits somewhere in the future.

Footnote 42 DAOD 3013-1 – Disposal of Surplus Materiel.

Footnote 43 For safety issues, weapons are treated like all other pieces of equipment, however ammunition safety issues are handled separately.

Footnote 44 DAOD 3002-1.

Footnote 45 Ammunition and Explosive Safety Manual, Volume 7.

Footnote 46 Data from DGLEPM Evaluation Questionnaire (N=136).

Footnote 47 It must be noted that the expression “ordering” used in this context does not necessarily mean “spending.” Many situations can take place between the moment the purchase order is printed out until the time it is processed, executed, paid out, and recorded as expenditure. However, the Financial Administration Act does not allow committing funds that are not accounted for. Therefore, the purchase orders have to be treated as potential expenditures.

Footnote 48 Source: DGMSSC. Distribution Resource Planning System.

Footnote 49 Source: DGMSSC. Distribution Resource Planning System.

Footnote 50 Source: DGMSSC. Distribution Resource Planning System.