Evaluation of the Sustainment Initiative

Back to Table of Contents

1.2 Executive Summary

This report presents the results of the evaluation of the Sustainment Initiative (SI), conducted during Fiscal Year (FY) 2019/20 by Assistant Deputy Minster (Review Services) (ADM(RS)) in compliance with the 2016 Treasury Board Policy on Results. The evaluation examines the performance of the SI over a four-year period, FY 2016/17 to FY 2019/20.

Sustainment Initiative Description and Objectives

Officially launched in 2016, the SI is a joint undertaking by the Department of National Defence (DND), Public Services and Procurement Canada (PSPC) and Innovation, Science and Economic Development Canada (ISED) with the overall goal of evolving the business of sustainment for military equipment and fleets. These departments work collaboratively to develop tailored sustainment solutions by using a Sustainment Business Case Analysis (SBCA) process aimed at optimizing four sustainment principles: performance, flexibility, economic benefit and value for money. The SBCA process requires early engagement and a whole-of-government approach to engaging stakeholders across both government and industry. The initiative is divided into five components, which include the three Equipment Program Management (EPM) divisions and the two Major Project Delivery (MPD) divisions. These cover the three environments (Air, Land and Maritime).

Scope

The evaluation examined relevance, effectiveness and efficiency of the SI’s activities. This evaluation concentrated on activities specific to DND; however, coordination and collaboration between the three departments was assessed. Industry representatives engaged during the SBCA process were also not included as an interview stakeholder group within the evaluation scope.

Key Findings

Relevance (continued need)

• The SI continues to address a demonstrable need for informed and optimal defence equipment sustainment solutions for the benefit of the Canadian Armed Forces (CAF) and Canadians.

Effectiveness

• Through the SBCA process and its Options Analysis phase, SI generally contributes to informed sustainment solutions that optimize the four sustainment principles.
• Practitioners require further support and training to further develop the necessary competencies.
• DND’s collaboration and coordination with the CAF and partner departments has improved. Industry collaboration is generally positive; however, there is some lack of understanding among practitioners and managers regarding industry’s role in the SBCA process.
• Efficiency-related concerns and communication challenges are hindering the extent to which practitioners and managers see value in the SBCA process.

Efficiency

• Individual roles and responsibilities in the SBCA process, as well as Centres of Expertise (CoE) and Other Government Departments (OGD), are generally understood; however, the Canadian defence industry’s roles and responsibilities are less clear.
• Gaps in SBCA accountability and inconsistent CoE delivery are hindering SI efficiency.
• CoEs, which guide SBCA teams through the SBCA process, and practitioners lack capacity, resources and flexibility to conduct SBCAs.
• SI’s draft Performance Measurement Framework (PMF) needs to be finalized and implemented to further assess impact.

Back to Table of Contents

Table 1. Summary of Key Findings and Recommendations

Key Finding	Recommendation
Relevance (continued need)
1. The SI continues to address a demonstrable need for informed and optimal defence equipment sustainment solutions for the benefit of the CAF and Canadians.
Effectiveness
2. Through the SBCA process and its Options Analysis phase, SI generally contributes to informed sustainment solutions that optimize the four sustainment principles.
3. SI has made progress in enhancing practitioners’ understanding of the SBCA process and principles; however, staff require further support and training to further develop the necessary competencies.	1. Explore options to address identified competency and training gaps.
4. DND’s collaboration and coordination efforts with the CAF and its partner departments (PSPC and ISED) have improved, although some challenges remain. Industry collaboration is generally positive; however, practitioners and managers would benefit from further clarity on the Canadian defence industry’s role in the SBCA process.	2. As SI matures, explore options for clarifying industry’s role in the SBCA process for DND practitioners/managers, as well as further engagement of the CAF in the SBCA process.
5. Efficiency-related concerns and communication challenges are hindering the extent to which practitioners and managers see value in the SBCA process.	3. Improve communication to address issues impacting practitioners’ and managers’ perception of SBCA value.
Efficiency
6. Despite some indications of SI efficiency, CoEs and practitioners lack capacity, resources and flexibility to conduct SBCAs.	4. Explore options for enhancing the efficiency of the SBCA process and ensure alignment of EPM/MPD divisions with SBCA workload.
7. While several improvements have been made to SI’s governance structure and execution, inconsistent CoE delivery across environments is posing some efficiency challenges, along with gaps in SBCA accountability and centralized oversight.	5. Clarify and formalize DND’s oversight on SBCAs, to ensure ongoing accountability and consistent CoE delivery where hindering efficiency.
8. Practitioners and their managers generally understand their individual roles and responsibilities in the SBCA process, as well as DND’s (including CoEs), the CAF representatives’ and Other Government Departments’; however, the Canadian defence industry’s roles and responsibilities are less clear.
9. SI recently developed a PMF, which needs to be refined and finalized.	6. Finalize and implement a PMF, with a view to assessing SBCA impact.

Back to Table of Contents

1.3 Program Profile (Description, Stakeholders)

DND’s EPM (Program 2.10) aims to ensure that the equipment required by the CAF to meet the training and readiness requirements is made available.

The SI is a component of the EPM. The SI is a joint undertaking by DND, PSPC and ISED with the overall goal of evolving the business of sustainment for military equipment and fleets.

Within the SI, the Sustainment Business Case Analysis (SBCA)process applies to every new or existing sustainment requirement valued at $20 million or more to the estimated end of life, and is aimed at developing tailored sustainment solutions that optimize the four sustainment principles, described adjacently.

The SBCA process requires early engagement and a whole-of-government approach to engaging stakeholders across both government and industry. The initiative is divided into five components, which include the three EPM programs and the two MPD divisions. These cover the three environments (Air, Land and Maritime).

Since 2016, 11 SBCA files have been completed, and there are currently 75 SBCA files in phases 1 or 2 of the SBCA process.

The Sustainment Principles

Performance

Is defence equipment operationally ready and mission capable?

Flexibility

Is the support system adaptable, scalable and readily adjusted to changes in operational requirements and/or operating budgets?

Economic Benefit

Are industrial benefits from defence procurements being leveraged to create jobs and economic growth for companies in Canada?

Value for Money

Are the required outcomes procured at a price commensurate with the market rate for comparable procurements?

Si Governance

DND Sustainment Director Generals (DG)

Assistant Deputy Minister (Materiel) (ADM(Mat)) management-level DGs involved in the SBCA reporting and approval process within the major projects and equipment programs, and in some cases attend DPS meetings.

Other government departments (OGD)

ISED and PSPC DGs are involved in the delivery and direction of SI; ISED and PSPC Directors have direct involvement in the SBCA process and SBCA files. Note that all three departments (DND, ISED and PSPC) participate in a DG-level SI Management Committee (SIMC), which focuses on management issues related to SI implementation, as well as an ADM-level SI Strategic Governance Committee (SISGC) which provides strategic direction for the successful implementation of SI.

DND SI Champion

This position is currently held at the Associate ADM level since January 2020. The role involves championing the SI by providing coherent oversight, and consistent direction and guidance. Sustainment DGs and Directors report to the SI Champion on the status of SBCAs.

SBCA support, professional development, communication and engagement

Sustainment Initiative Program Management (SIPM) and Directorate Materiel Policy & Procedure 9 (DMPP)

ADM(Mat) Directorate responsible for the development, implementation and coordination of SI, including policies, procedures and training resources.

SBCA Support

Centres of Expertise (CoE)

The CoEs guide SBCA teams through the SBCA process and are expected to play an active role within SBCA teams. Each environment has its own CoE (Air, Land and Maritime).

SBCA support and conduct

DND Sustainment and Procurement Directors

ADM(Mat) Directors involved in the SBCA reporting and approval process within the major projects and equipment programs.

SBCA conduct

DND practitioners and managers

Primarily technical and procurement personnel working on SBCAs within the ADM(Mat) major projects and equipment programs.

CAF representatives

CAF personnel are involved in the development of sustainment requirements (phase 1B) of the SBCA process, and are the end-users of the equipment.

Other stakeholder groups not included in the scope of the evaluation include industry representatives that are engaged during the SBCA process. Additionally, note that the OGD stakeholder group also participates in SBCA support and conduct.

Back to Table of Contents

2. Continued need

2.1 Finding

There is a continued need for the SI.

Need for Sustainment

Documentary evidence suggests that trends in state and costs of maintaining equipment/fleets are rising due to aging fleets and increasingly complex equipment, with sustainment and operations costs over the lifecycle of major systems equating to more than twice the acquisition costs.

Approximately $3.2 billion of the Defence Budget was allocated to equipment sustainment in 2019/20.

Additionally, the number of SBCA files has increased significantly since 2016, suggesting an increase in SBCA workload and a continued trend in rising demand for sustainment.

CAF Operational Readiness

SI supports CAF operational readiness and the CAF mandate, to ensure evidence-based decision making and maintain a military that is “agile, multi-purpose and combat-ready, operated by highly trained, well-equipped women and men.”

"SI is a continuous requirement since there will always be new equipment, new capability requirements, and equipment to sustain.”

- interview, DND sustainment DG

Accountability

Justification of decision making and resource responsibility or stewardship were the main themes discussed by interviewees regarding the importance of SI need, and were described as practices that were not observed or considered sufficiently in the past. These themes were also noted in the 2011 and 2016 Office of the Auditor General audit reports.

Alignment with Federal Initiatives and Policies

SI aligns with federal strategic initiatives and policies, a key one being the DPS. The DPS is a government-wide initiative that applies to all defence procurements valued at $20 million or more, and is guided by the following principles: whole-of- government collaboration; early and regular stakeholder and industry engagement; and timely and effective decision making. DPS represents the Government of Canada’s commitment to ensuring that purchases of Defence equipment and services result in job creation and growth for Canadian industry. It builds on key lessons learned from the development of the National Shipbuilding Strategy which introduced a new model for managing major procurements and leveraging benefits to Canada, as well as on key principles from PSPC’s Smart Procurement approach, which also includes benefits to Canadians.

Back to Table of Contents

3. Effectiveness

3.1 Findings

The SBCA process supports the development of informed sustainment solutions and optimized principles.

Due to few SBCAs having been fully completed, the evaluation examined the extent to which the SI/SBCA process positions itself for future success through the effective development of sustainment requirements in phase 1, and the effective completion of the Options Analysis in phase 2.

Informed Sustainment Solutions

Overall, evaluation evidence indicates that the SI/SBCA process contributes positively to developing informed tailored sustainment solutions. The SBCA process and specifically the Options Analysis phase provides a framework of analysis that encourages consideration of all sustainment principles and multiple sustainment options. In following this framework, interview, document review and survey evidence suggest that the SBCA process in general contributes to the development of informed sustainment solutions, encouraging DND SBCA practitioners to think critically and collaborate.

Optimized Sustainment Principles

Evaluation evidence indicates that the SI/SBCA process contributes to optimized principles. The following SBCA examples demonstrate the outcome of optimized sustainment principles, linked to the primary areas impacted:

Table 3. Evidence that the SI/SBCA process contributes to optimized principles.

Performance	The calibration services SBCA reduced the turnaround time of services from a month to an average of ten days through contracting of increased number of commercial facilities, and reduced costs and incentivized efficiency by transferring transportation responsibility to suppliers.
Flexibility	The T56 and F404 engines SBCA incentivized contractors to maximize use of inventory, eliminating many redundant parts and reducing dormant stock.
Economic Benefit	The MX-15 Sky Cam SBCA transferred life-cycle supports supplied by the original equipment manufacturer in- house to DND, and transportation logistics of turrets was contracted out, reducing overall delivery times.
Value for Money	The CC-150 aircraft SBCA Statement of Work and modified 5-year option period contract, incentivized contractors to increase aircraft availability and deliver services in a financially responsible manner.

Back to Table of Contents

Further support and training is needed to develop practitioner competencies required for optimal sustainment solutions.

Impact of SI Resources

When asked about the impact of support resources on increasing their understanding of how to conduct an SBCA, practitioners and managers noted that SI support was impactful overall; however, CoE support had the greatest impact. Interview evidence corroborates that CoE support has made valuable contributions in mitigating competency gaps of practitioners.

Understanding of SBCA Process and Principles

Managers’ and practitioners' understanding of sustainment principles as well as the SBCA process was generally good or very good, as seen in the following figure.

When examined with consideration of the 2017 Sustainment Initiative Health Check (SIHC) survey findings, evaluation survey evidence suggests SI has made progress towards improving practitioners’ understanding of the SBCA process. Survey respondents also indicated a gradual decline in comprehension of the SBCA process when nearing Phase 3, potentially due to SI’s infancy.

Knowledge and Skills

As demonstrated in the following graph, practitioners and managers were most confident in the areas of teamwork, familiarization with sustainment concepts and critical thinking, and less confident in creativity/innovation, business acumen and data analytics.

Some interviewees indicated that skills related to business acumen and innovation cannot be taught and must be acquired through experience, which may explain the lower levels of confidence exhibited by survey respondents in these areas.

21% of survey respondents that offered comments to improve the SI/SBCA process suggested increased training for practitioners; and

24% suggested simplifying/clarifying the SBCA guide.

When asked about practitioner knowledge and skills, interviewees indicated that skills were insufficient in areas such as Performance-based Contracting, business acumen and innovation. However, many interviewees found it challenging to comment on practitioner knowledge and skills due to limitations in resources, time and SBCA process flexibility which impact practitioner learning and development. In a 2018/2019 course evaluation report, 93% of respondents felt that the knowledge and skills presented during the Introduction to SBCA course would improve their work; however, only 27% felt they would be able to apply their learning into practice, suggesting a gap between theory and application. Addressing these shortcomings would contribute to the development of optimal sustainment solutions.

Back to Table of Contents

3.2 Recommendations

Collaboration and coordination efforts directed towards internal stakeholders have improved.

DND/CAF Collaboration

Opinions are mixed among stakeholders regarding the effectiveness of the CAF’s input into the SBCA process. DGs and directors noted that CAF input has been beneficial in defining requirements and conducting options analysis; however, SIPM and CoE interviewees noted that the CAF does not always fully understand the purpose and objectives of SI. This presents an opportunity for further integration between DND/CAF stakeholders.

Overall, survey respondents are satisfied with DND/CAF and OGD collaboration, and are less satisfied with industry collaboration.

OGD Collaboration

Collaboration between the three departments has improved, and is perceived as working well overall. Collaboration between PSPC and DND is functioning well at the director level; however, some challenges were noted at the working level, particularly conflicts regarding SBCA leadership between Contracting Authority and Technical Authority roles.

Collaboration with Industry

Practitioners and managers generally believe industry is being sufficiently engaged. Responses were slightly less positive with respect to the effectiveness of input being received from industry.

57% of survey respondents say industry engagement is at least somewhat sufficient; and

46% say input from industry is at least somewhat effective.

Similar to the industry collaboration results, a notable percentage of survey responses fell within the do not know or no response category (22% for industry engagement, 29% for industry input), which suggests a lack of understanding among practitioners and managers regarding industry’s role in the SBCA process. Additional key evidence related to this is discussed in slide 16. Finally, the majority of DND DGs and directors interviewed agreed that industry produced beneficial input/feedback. This suggests that engagement is more effective at the senior level than at the working level.

Practitioners’ and managers’ perception of the SBCA process value could be improved.

Since 2016, efforts have been made to convey the initiative’s value through courses, workshops, announcements, materials and websites. Many interviewees noted that SI involves significant culture change, and that perceived value will increase as more SBCAs are fully completed and further benefits are identified.

Perception of SBCA process value

Evaluation evidence indicates that management generally sees value in the SBCA process, noting increased analytical rigor as an important benefit; however, reactions are more mixed at the working level. When practitioners and managers were asked to comment on which aspects of the SBCA process are/are not worthwhile, 18% noted positive aspects (e.g., allows DND to be a “smart consumer”); however, a number of efficiency-related concerns were raised, which are discussed in further detail in slide 14.

Communication challenges are also impacting the extent to which stakeholders see value in the SBCA process. Gaps in understanding of SBCA applicability have led some to question SBCA value, noting that SBCAs are “not relevant” for all sustainment solutions/contracts. Misconceptions regarding SI objectives were also identified, as a number of stakeholders viewed the primary goal of SI as cost savings versus optimization of all four sustainment principles. There also appeared to be a lack of awareness of recently achieved results, as many stakeholders were looking for more current, concrete results despite SIPM’s recent identification and communication of these.

Survey results suggest there is an opportunity to increase practitioners’ and managers’ perceived value of the SBCA process.

While 36% of responses were positive (“quite a bit” or “a great deal”), over half (54%) of responses fell within the “somewhat” or “very little” categories.

Results regarding the CAF’s perspective on SBCA process value were mixed. The CAF representative interviewees generally perceive the SBCA process as valuable; however, other (DND management) interviewees noted that SBCAs are not a focus for the CAF, nor is their importance well understood.

Back to Table of Contents

4. Efficiency

4.1 Findings

Efficiency gaps remain in the SBCA process, and there is insufficient capacity to conduct SBCAs.

SI Efficiency

Since 2016, SI has demonstrated consistent efforts to enhance its efficiency. Examples include the development of a standardized process of engagement for the kick-off meeting in phase 1A of the SBCA process requiring all stakeholders to participate, streamlining the SBCA approval process, and the creation of a lessons learned repository to support practitioners working on SBCAs and development of best practices.

Overall, DND budget and actual expenditures for SI (DMPP-9 and CoEs) resources and personnel from FY 2016/17 to 2019/20 aligned*. Further, as noted on slide 16, individual roles and responsibilities in the SBCA process, as well as CoEs and OGDs, are generally understood.

Capacity and Resources

Gaps in CoE capacity were first identified in the 2017 Health Check, and SI management has worked towards addressing this.

CoEs have a considerable impact on SI and SBCAs; however, as noted by a former Sustainment DG in June 2019, SI needs to continue “leaning forward with staffing” and “cannot wait until SBCA work load exceeds capacity to find a solution.”

52% of SBCA supervisors reported that DND SBCA practitioners have insufficient human resources to conduct SBCAs.

About half of the practitioners (49%) as well as their supervisors (51%) reported that practitioners have insufficient time to conduct SBCAs, and when asked, process length and complexity were in their top three concerns regarding efficiency.

Training gaps (discussed earlier in slide 11) were also noted in survey and interview results (Sustainment Directors) as impacting efficiency. SIPM indicated that more guidance and tools for practitioners would benefit SBCA efficiency.

All sustainment and procurement directors interviewed emphasized capacity/resource limitations as the primary efficiency challenge. Sustainment DG interviewees also noted that the National Procurement budget is divided across environments and split between operations/maintenance and procurement, leaving little leeway.

SBCA Process Flexibility

All sustainment directors and several CAF interviewees indicated that the SBCA process would benefit from fewer and/or streamlined steps, increased clarity and reconsideration of timelines. However, SIPM/DMPP indicated that streamlining is already built into the SBCA process, and “people just don’t know it.”

34% of SBCA Practitioners and Managers find the SBCA process is too long.

24% find the SBCA process too complex.

4.2 Recommendations

*SIPM was unable to provide the information required during the course of the evaluation.

Gaps in SBCA accountability and inconsistent CoE delivery are hindering SI efficiency.

CoE Delivery

Evidence suggests that inconsistencies in CoE execution has led to some confusion regarding the CoE role among stakeholders. Inconsistencies were identified regarding the CoEs’ approach to supporting SBCAs, with some taking a more prescriptive approach than others, as well as when SBCAs were being applied. The level of CoE support and CoE involvement also varies considerably, with the Air CoE having more capacity and taking a more involved role. This approach aligns with the 2019 CoE TOR, which calls for active involvement throughout the SBCA process, as well as feedback from practitioner and manager survey respondents who expressed a need for additional CoE guidance.

22% of interviewees and 19% of survey comment respondents noted inconsistent CoE delivery.

CoE interviewee responses were mixed with regards to the appropriate level of support for SBCA teams and whether there should be alignment between environments. While some CoEs advocated for active involvement, others supported a more distanced approach to encourage greater ownership of SBCAs.

Top themes across survey respondents that identified CoE delivery inconsistencies included:

CoE current role, akin to a gatekeeper or challenge function, was perceived as impeding the efficiency of the SBCA process.

Governance/SBCA Accountability

Since the 2017 SIHC, adjustments to the SI governance framework have been completed, a key one being updating the TORs for SIPM, SIMC and SISGC in 2018, and CoE in 2019. However, SIPM and CoE interviewees indicated a need for greater SBCA accountability at the DG level and higher to ensure consistency in SBCA execution, direction and messaging. DND sustainment DG interviewees also noted the need for more rigorous oversight from leadership. Until recently, there was no central oversight in place to ensure accountability for and to monitor the status of SBCAs and timelines across EPMs and MPDs.

11% of survey respondents who commented on suggestions for improvement noted leadership accountability as an area for potential improvement.

Back to Table of Contents

There is room for improvement with respect to the roles and responsibilities of the Canadian defence industry.

Individual roles and responsibilities

Survey results indicate that DND SBCA practitioners and managers sufficiently understand their roles and responsibilities related to conducting or supporting the SBCA process.

The majority of surveyed practitioners and managers have a good or very good understanding of their roles and responsibilities.

Roles and responsibilities of partners

Survey results indicate that understanding of the roles and responsibilities of partners involved in the SBCA process is good overall. Canadian defence industry’s roles and responsibilities were less understood.

Back to Table of Contents

SI’s draft PMF needs to be finalized and implemented to further assess impact.

Performance Measurement Progress

Interview and documentary evidence indicates that SIPM has had some challenges in developing appropriate performance measurement metrics. Interviewees noted difficulties in commenting on the quality of SBCA solutions due to the lack of an implemented PMF; however, they also acknowledged the challenge in doing so as each sustainment solution is different.

SIPM recently established a draft PMF that includes SI and SBCA-level metrics, which was approved by SISGC in 2019 with some feedback to be addressed. SIPM also provides some guidance on performance measurement in the options analysis phase of the SBCA Guide and indicates that performance strategies are to be implemented at the sustainment solution monitoring phase.

T56+F404 Engine SBCA Example

• Close to 100% performance targets being met
• Payments based on completion rather than progress
• Flexibility integrated to adapt to changing RCAF requirements

3 out of 5

DND sustainment directors noted the absence of common metrics as a challenge to measuring the success of the SI.

Planned Performance Measurement

SIPM’s draft PMF is well-positioned as it assesses the quality of SBCA process itself (i.e., ensures operational sustainment requirements are sufficiently defined) – which is the foundation for developing optimal sustainment solutions. It also ensures a well-defined PMF structure will be put in place that can be supported by practical data sources.

The PMF also supports the assessment of SBCA “impact” by assessing performance against the four principles, with specific performance indicators to be developed and collected by SBCA teams and reported annually. With SI having been implemented for four years, it is important that performance information be collected systematically to support the ongoing assessment of SBCA/SI performance, as well as to support decision making and program improvements.

Finally, the PMF’s SI-level metrics are useful as “health indicators”; however, they are limited in the extent to which they can be attributed to SI. For example, several factors external to SI may contribute to ADM(Mat)’s “fleet availability,” which is the current performance measurement for SI’s “performance” principle.

Back to Table of Contents

5. Conclusion & annexes

5.1 Conclusions

DND has made progress towards achieving its intended SI outcomes since 2016. Key accomplishments include increased practitioners’ and managers’ understanding of the SBCA process and sustainment principles; improved collaboration and coordination with key stakeholders; and early indications of optimized sustainment principles. However, efficiency-related issues - particularly the lack of CoE and practitioner resources to conduct SBCAs - are hindering DND’s success. Gaps in communication of SBCA process value, practitioner training and support, and clarity of the Canadian defence industry’s role are also limiting SI’s ability to achieve its outcomes. Finally, SI’s PMF needs to be finalized and implemented to measure SI’s success in the near future.

Back to Table of Contents

5.2 Annex A - Management Action Plan

Back to Table of Contents

5.3 Annex B - Evaluation Scope

Coverage and Responsibilities

Due to the SI’s recent implementation (officially launched in 2016), the evaluation covers the four-year period from FY 2016/17 to FY 2019/20.

The evaluation examines the relevance (i.e., continued need), effectiveness and efficiency of the SI’s activities. This evaluation concentrates on activities specific to DND, as PSPC’s and ISED’s activities were considered outside of scope. However, coordination and collaboration between the three departments was assessed. Additional information on evaluation approach, methodology and limitations can be found in Annex D.

Context

Between March and June 2017, an SIHC was conducted by ADM(Mat) staff with support from ADM(RS). Its objective was to assess the progress made in implementing SI principles and process across all three departments. As SI was still in its infancy, the SIHC assessed the extent to which the foundational aspects were in place to enable future and enduring success. Where applicable, the evaluation used the SIHC report as a baseline, to assess SI’s progress towards achieving outcomes since 2017.

In January 2018, a Joint Statement to Practitioners noted that SI had completed the phase-in period, which consisted of a number of pilot projects and 20 SBCAs. That year, the examination of SI’s benefits for DND/CAF was requested by the 2018 ADM(Mat).

Back to Table of Contents

5.4 Annex C - Program Profile (Key Activities)

Key SI activities include the following:

1. SBCA Conduct and Support

DND sustainment practitioners conduct SBCAs with guidance, coaching and mentoring provided by interdepartmental CoEs.

SI offers the following primary support resources: an SBCA Guide explaining the process; a GCpedia webpage explaining the initiative; an online introduction to SI course; a three-day in-person introduction to SBCA workshop; an Executive workshop; and a CoE tri-departmental team for each environment.

2. Policy and Procedures

Conduct continuous policy and process improvements (e.g., SBCA Guide and support procedural document revisions and enhancements).

3. Resourcing and Professional Development

Assign resources to conduct SBCAs; provide professional development/training opportunities (courses, workshops)

4. Communication and Engagement

Develop and implement communication and engagement products/strategies - internally within DND as well as with external stakeholders (e.g., industry).

5. Governance

Develop and maintain SI leadership and oversight within DND (SI Program Management) and across the three departments (i.e., DG-level SIMC, ADM-level SISGC).

Activities related to sustainment include, but are not limited to: maintenance, repair and overhaul; engineering support; engineering test and evaluation; maintenance program management; fleet management activities; acquisition of parts; logistics management; and sustainment-related training.

Back to Table of Contents

5.5 Annex D - Evaluation Approach, Methodology and Limitations

Evaluation Approach

Due to the initiative being in the early stages of implementation, the evaluation focused on the assessment of short-term outcomes, and developed forward-looking indicators to inform on SI’s current and planned development and implementation.

Immediate Outcomes:

• Increased competencies to develop tailored sustainment solutions;
• Improved collaboration and coordination among stakeholders; and
• Increased recognition of the value of the SBCA among stakeholders.

Intermediate Outcome:

• Development of informed, tailored sustainment solutions that optimize performance, value for money, flexibility and economic benefits.

Data Collection Methods

The evaluation used multiple lines of evidence and complementary qualitative and quantitative research methods (described in more detail in the following slide) to help ensure the reliability of information and data supporting findings:

• key informant interviews;
• online survey;
• document review;
• financial/Human Resources (HR) data analysis; and
• direct observation.

Data Collection Methods

The lines of evidence collected through the conduct phase of the evaluation were triangulated with each other and verified with program officials to ensure their validity. The research methodology used in the scoping and conduct of the evaluations are as follows:

Key Informant Interviews

The evaluation team worked with SIPM (and CoE leads in some cases) to identify interviewees and receive feedback on the interview guide. There were a total of 31 interviews conducted with the key stakeholder groups outlined in the following table. The evaluation team worked with CoE leads to ensure sufficient representation was met for the three environments, and selected interviewees who had the most experience conducting SBCAs, and who had advanced the furthest in the process (i.e., phase 3).

Table D-1. Key Informant Interviews (31 meetings total)

Internal to DND/CAF:
CoE (1 interview/environment)	n=3
DND Sustainment Directors	n=6
DND Sustainment DGs	n=5
DND Procurement Directors	n=2
CAF representatives	n=5
SIPM/DMPP	n=6
DND Associate ADM/SI Champion	n=1
External to DND/CAF:
OGDs (PSPC and ISED)	n=3

Online Survey

An online survey was developed for DND/CAF sustainment practitioners and managers/supervisors who are involved in SI. It was designed in consultation with SIPM and CoE leads. The survey focused on assessing the initiative’s performance (effectiveness and efficiency). It was launched over a two-week period (June 1– June 12, 2020), and distributed using a Chain of Command approach, initiated from the Associate ADM (SI Champion) down to the working level. Overall, there was a response rate of ~ 50% (208/~400); however, only 160 respondents indicated their involvement in conducting/support SBCAs, which was a key condition of the evaluation. Thus, only responses from the noted 160 participants were used in the analysis phase of the evaluation. Indications from SIPM suggest that adequate representation was met across the five applicable groups (Director General Land Equipment Program Management, Director General Maritime Equipment Program Management, Director General Aerospace Equipment Program Management, Director General Major Project Delivery (Air and Land), Director General Major Project Delivery (Sea)).

Document Review

A preliminary review of foundational documents was conducted during the planning phase of the evaluation to ascertain a fulsome understanding of the program to inform the development of the scope, logic model and evaluation matrix. An official Request for Information was then submitted to SIPM to obtain all pertinent documentation that would assist in the program assessment. The evaluation team received over 100 documents, which included: government websites; departmental administrative reports; program documents, some draft, others final; external reports; and policies and mandates applicable to the program.

Financial/HR Data Analysis

A review of financial and HR data was conducted to assess the efficiency of the program since its official launch in 2016. Information was obtained from the three CoEs, DMPP and SIPM.

Direct Observation

A member of the evaluation team attended an executive SBCA workshop led by the program to observe the method of teaching, and level of engagement/feedback of participants. No other opportunities presented themselves due to limitations resulting from the pandemic.

Evaluation Limitations

The limitations encountered by the evaluation and mitigation strategies employed in the evaluation process are outlined in the following table.

Table D-2. Evaluation Limitations

	Limitations	Mitigation Strategies
External Factors	SI is a tri-departmental initiative, thus, a DND- focused evaluation may not capture all aspects of the program’s relevance, effectiveness and efficiency.	The evaluation team focused on the achievement of outcomes prioritized by DND, specifically, and has put forward recommendations that reflect DND’s scope of influence.
Nascent Program	The initiative is in the early stages of implementation. Therefore, the evaluation’s scope was limited to assessing short-term outcomes.	To the extent possible, the evaluation team incorporated forward-looking indicators to inform on SI’s current and planned development and implementation.
Data Availability	Limited performance data/information available (e.g., PMF has not been implemented).	The evaluation relied more heavily on the following lines of evidence to inform the findings and recommendations: key informant interviews; a survey; financial/HR analysis; direct observation; and document review.
Qualitative vs. Quantitative	Due to the nature of the outcomes on which the evaluation focused, there is a potential risk that the evidence obtained may be largely perception-based.	This limitation was addressed by triangulating evidence from multiple data sources to inform the findings and ensuing recommendations (e.g., interviews from stakeholders at varying levels and with a range of experience across DND/CAF, review of pertinent program documentation, financial/HR data analysis, direct observation of program participants, and a survey to assess the experiences of working-level staff).
Global Pandemic	Unforeseen change in working environment due to a global pandemic could inhibit the evaluation team’s ability to complete the project on time.	Evaluation team members were able to make use of collaborative tools to move forward with the conduct, analysis and report writing phases of the project, while working remotely.

Back to Table of Contents