Evaluation of the Sustainment Initiative
- Assistant Deputy Minister (Materiel)
- Assistant Deputy Minister (Review Services)
- Canadian Armed Forces
- Centre of Expertise
- Director General
- Director, Materiel Policy and Procedures
- Department of National Defence
- Defence Procurement Strategy
- Equipment Program Management
- Fiscal Year
- Human Resources
- Innovation, Science and Economic Development Canada
- Major Project Delivery
- Other Government Department
- Office of Primary Interest
- Performance Measurement Framework
- Public Services and Procurement Canada
- Sustainment Business Case Analysis
- Sustainment Initiative
- Sustainment Initiative Health Check
- Sustainment Initiative Management Committee
- Sustainment Initiative Program Management
- Sustainment Initiative Strategic Governance Committee
- Terms of Reference
This report presents the results of the evaluation of the Sustainment Initiative (SI), conducted during Fiscal Year (FY) 2019/20 by Assistant Deputy Minster (Review Services) (ADM(RS)) in compliance with the 2016 Treasury Board Policy on Results. The evaluation examines the performance of the SI over a four-year period, FY 2016/17 to FY 2019/20.
Sustainment Initiative Description and Objectives
Officially launched in 2016, the SI is a joint undertaking by the Department of National Defence (DND), Public Services and Procurement Canada (PSPC) and Innovation, Science and Economic Development Canada (ISED) with the overall goal of evolving the business of sustainment for military equipment and fleets. These departments work collaboratively to develop tailored sustainment solutions by using a Sustainment Business Case Analysis (SBCA) process aimed at optimizing four sustainment principles: performance, flexibility, economic benefit and value for money. The SBCA process requires early engagement and a whole-of-government approach to engaging stakeholders across both government and industry. The initiative is divided into five components, which include the three Equipment Program Management (EPM) divisions and the two Major Project Delivery (MPD) divisions. These cover the three environments (Air, Land and Maritime).
The evaluation examined relevance, effectiveness and efficiency of the SI’s activities. This evaluation concentrated on activities specific to DND; however, coordination and collaboration between the three departments was assessed. Industry representatives engaged during the SBCA process were also not included as an interview stakeholder group within the evaluation scope.
DND has made progress in achieving intended SI outcomes since 2016. Key accomplishments include: increased practitioners’ and managers’ understanding of the SBCA process and sustainment principles; improved collaboration and coordination with key stakeholders; and early indications of optimized sustainment principles. Efficiency-related issues are hindering DND’s success. CoEs and practitioners lack resources to conduct SBCAs; gaps in communication of SBCA process value, practitioner training and support, and clarity of Canadian defence industry’s role were also identified. Finally, SI’s draft PMF needs to be finalized and implemented to measure SI’s success in the near future.
Relevance (continued need)
- The SI continues to address a demonstrable need for informed and optimal defence equipment sustainment solutions for the benefit of the Canadian Armed Forces (CAF) and Canadians.
- Through the SBCA process and its Options Analysis phase, SI generally contributes to informed sustainment solutions that optimize the four sustainment principles.
- Practitioners require further support and training to further develop the necessary competencies.
- DND’s collaboration and coordination with the CAF and partner departments has improved. Industry collaboration is generally positive; however, there is some lack of understanding among practitioners and managers regarding industry’s role in the SBCA process.
- Efficiency-related concerns and communication challenges are hindering the extent to which practitioners and managers see value in the SBCA process.
- Individual roles and responsibilities in the SBCA process, as well as Centres of Expertise (CoE) and Other Government Departments (OGD), are generally understood; however, the Canadian defence industry’s roles and responsibilities are less clear.
- Gaps in SBCA accountability and inconsistent CoE delivery are hindering SI efficiency.
- CoEs, which guide SBCA teams through the SBCA process, and practitioners lack capacity, resources and flexibility to conduct SBCAs.
- SI’s draft Performance Measurement Framework (PMF) needs to be finalized and implemented to further assess impact.
Relevance (continued need)
1. The SI continues to address a demonstrable need for informed and optimal defence equipment sustainment solutions for the benefit of the CAF and Canadians.
2. Through the SBCA process and its Options Analysis phase, SI generally contributes to informed sustainment solutions that optimize the four sustainment principles.
3. SI has made progress in enhancing practitioners’ understanding of the SBCA process and principles; however, staff require further support and training to further develop the necessary competencies.
1. Explore options to address identified competency and training gaps.
4. DND’s collaboration and coordination efforts with the CAF and its partner departments (PSPC and ISED) have improved, although some challenges remain. Industry collaboration is generally positive; however, practitioners and managers would benefit from further clarity on the Canadian defence industry’s role in the SBCA process.
2. As SI matures, explore options for clarifying industry’s role in the SBCA process for DND practitioners/managers, as well as further engagement of the CAF in the SBCA process.
5. Efficiency-related concerns and communication challenges are hindering the extent to which practitioners and managers see value in the SBCA process.
|3. Improve communication to address issues impacting practitioners’ and managers’ perception of SBCA value.|
6. Despite some indications of SI efficiency, CoEs and practitioners lack capacity, resources and flexibility to conduct SBCAs.
4. Explore options for enhancing the efficiency of the SBCA process and ensure alignment of EPM/MPD divisions with SBCA workload.
7. While several improvements have been made to SI’s governance structure and execution, inconsistent CoE delivery across environments is posing some efficiency challenges, along with gaps in SBCA accountability and centralized oversight.
5. Clarify and formalize DND’s oversight on SBCAs, to ensure ongoing accountability and consistent CoE delivery where hindering efficiency.
8. Practitioners and their managers generally understand their individual roles and responsibilities in the SBCA process, as well as DND’s (including CoEs), the CAF representatives’ and Other Government Departments’; however, the Canadian defence industry’s roles and responsibilities are less clear.
9. SI recently developed a PMF, which needs to be refined and finalized.
6. Finalize and implement a PMF, with a view to assessing SBCA impact.
Table 1 Details - Summary of Key Findings and Recommendations.
DND’s EPM (Program 2.10) aims to ensure that the equipment required by the CAF to meet the training and readiness requirements is made available.
The SI is a component of the EPM. The SI is a joint undertaking by DND, PSPC and ISED with the overall goal of evolving the business of sustainment for military equipment and fleets.
Within the SI, the Sustainment Business Case Analysis (SBCA)process applies to every new or existing sustainment requirement valued at $20 million or more to the estimated end of life, and is aimed at developing tailored sustainment solutions that optimize the four sustainment principles, described adjacently.
The SBCA process requires early engagement and a whole-of-government approach to engaging stakeholders across both government and industry. The initiative is divided into five components, which include the three EPM programs and the two MPD divisions. These cover the three environments (Air, Land and Maritime).
Since 2016, 11 SBCA files have been completed, and there are currently 75 SBCA files in phases 1 or 2 of the SBCA process.
Figure 1 Details - SBCA Process Guide.
The Sustainment Principles
Is defence equipment operationally ready and mission capable?
Is the support system adaptable, scalable and readily adjusted to changes in operational requirements and/or operating budgets?
Are industrial benefits from defence procurements being leveraged to create jobs and economic growth for companies in Canada?
Value for Money
Are the required outcomes procured at a price commensurate with the market rate for comparable procurements?
Within the SI, the Sustainment Business Case Analysis (SBCA) process applies to every new or existing sustainment requirement valued at $20 million or more to the estimated end of life, and is aimed at developing tailored sustainment solutions that optimize the four sustainment principles, described adjacently.
DND Sustainment Director Generals (DG)
Assistant Deputy Minister (Materiel) (ADM(Mat)) management-level DGs involved in the SBCA reporting and approval process within the major projects and equipment programs, and in some cases attend DPS meetings.
Other government departments (OGD)
ISED and PSPC DGs are involved in the delivery and direction of SI; ISED and PSPC Directors have direct involvement in the SBCA process and SBCA files. Note that all three departments (DND, ISED and PSPC) participate in a DG-level SI Management Committee (SIMC), which focuses on management issues related to SI implementation, as well as an ADM-level SI Strategic Governance Committee (SISGC) which provides strategic direction for the successful implementation of SI.
DND SI Champion
This position is currently held at the Associate ADM level since January 2020. The role involves championing the SI by providing coherent oversight, and consistent direction and guidance. Sustainment DGs and Directors report to the SI Champion on the status of SBCAs.
SBCA support, professional development, communication and engagement
Sustainment Initiative Program Management (SIPM) and Directorate Materiel Policy & Procedure 9 (DMPP)
ADM(Mat) Directorate responsible for the development, implementation and coordination of SI, including policies, procedures and training resources.
Centres of Expertise (CoE)
The CoEs guide SBCA teams through the SBCA process and are expected to play an active role within SBCA teams. Each environment has its own CoE (Air, Land and Maritime).
SBCA support and conduct
DND Sustainment and Procurement Directors
ADM(Mat) Directors involved in the SBCA reporting and approval process within the major projects and equipment programs.
DND practitioners and managers
Primarily technical and procurement personnel working on SBCAs within the ADM(Mat) major projects and equipment programs.
CAF personnel are involved in the development of sustainment requirements (phase 1B) of the SBCA process, and are the end-users of the equipment.
Other stakeholder groups not included in the scope of the evaluation include industry representatives that are engaged during the SBCA process. Additionally, note that the OGD stakeholder group also participates in SBCA support and conduct.
There is a continued need for the SI.
Finding 1: The SI continues to address a demonstrable need for informed and optimal defence equipment sustainment solutions for the benefit of the CAF and Canadians.
Need for Sustainment
Documentary evidence suggests that trends in state and costs of maintaining equipment/fleets are rising due to aging fleets and increasingly complex equipment, with sustainment and operations costs over the lifecycle of major systems equating to more than twice the acquisition costs.
Approximately $3.2 billion of the Defence Budget was allocated to equipment sustainment in 2019/20.
Additionally, the number of SBCA files has increased significantly since 2016, suggesting an increase in SBCA workload and a continued trend in rising demand for sustainment.
Figure 2 Details - SBCAs workload indicating a continued trend in rising demand for sustainment (2016-2020).
CAF Operational Readiness
SI supports CAF operational readiness and the CAF mandate, to ensure evidence-based decision making and maintain a military that is “agile, multi-purpose and combat-ready, operated by highly trained, well-equipped women and men.”
"SI is a continuous requirement since there will always be new equipment, new capability requirements, and equipment to sustain.”
- interview, DND sustainment DG
Justification of decision making and resource responsibility or stewardship were the main themes discussed by interviewees regarding the importance of SI need, and were described as practices that were not observed or considered sufficiently in the past. These themes were also noted in the 2011 and 2016 Office of the Auditor General audit reports.
Alignment with Federal Initiatives and Policies
SI aligns with federal strategic initiatives and policies, a key one being the DPS. The DPS is a government-wide initiative that applies to all defence procurements valued at $20 million or more, and is guided by the following principles: whole-of- government collaboration; early and regular stakeholder and industry engagement; and timely and effective decision making. DPS represents the Government of Canada’s commitment to ensuring that purchases of Defence equipment and services result in job creation and growth for Canadian industry. It builds on key lessons learned from the development of the National Shipbuilding Strategy which introduced a new model for managing major procurements and leveraging benefits to Canada, as well as on key principles from PSPC’s Smart Procurement approach, which also includes benefits to Canadians.
The SBCA process supports the development of informed sustainment solutions and optimized principles.
Finding 2: Through the SBCA process and its Options Analysis phase, SI generally contributes to informed sustainment solutions that optimize the four sustainment principles.
Due to few SBCAs having been fully completed, the evaluation examined the extent to which the SI/SBCA process positions itself for future success through the effective development of sustainment requirements in phase 1, and the effective completion of the Options Analysis in phase 2.
Informed Sustainment Solutions
Overall, evaluation evidence indicates that the SI/SBCA process contributes positively to developing informed tailored sustainment solutions. The SBCA process and specifically the Options Analysis phase provides a framework of analysis that encourages consideration of all sustainment principles and multiple sustainment options. In following this framework, interview, document review and survey evidence suggest that the SBCA process in general contributes to the development of informed sustainment solutions, encouraging DND SBCA practitioners to think critically and collaborate.
Figure 3 Details - Survey results indicate that the SI/SBCA process contributes positively to developing informed tailored sustainment solutions.
Optimized Sustainment Principles
Evaluation evidence indicates that the SI/SBCA process contributes to optimized principles. The following SBCA examples demonstrate the outcome of optimized sustainment principles, linked to the primary areas impacted:
Value for Money
Table 3 Details - Evidence that the SI/SBCA process contributes to optimized principles.
Further support and training is needed to develop practitioner competencies required for optimal sustainment solutions.
Finding 3: SI has made progress in enhancing practitioners’ understanding of the SBCA process and principles; however, staff require further supportand training to further develop the necessary competencies for developing optimal sustainment solutions.
Impact of SI Resources
When asked about the impact of support resources on increasing their understanding of how to conduct an SBCA, practitioners and managers noted that SI support was impactful overall; however, CoE support had the greatest impact. Interview evidence corroborates that CoE support has made valuable contributions in mitigating competency gaps of practitioners.
Understanding of SBCA Process and Principles
Managers’ and practitioners' understanding of sustainment principles as well as the SBCA process was generally good or very good, as seen in the following figure.
Figure 4 Details - 70% of respondents indicated having a good to very good understanding of the sustainment principles.
Figure 5 Details - 75% of respondents indicated having a good to very good understanding of the SBCA process.
When examined with consideration of the 2017 Sustainment Initiative Health Check (SIHC) survey findings, evaluation survey evidence suggests SI has made progress towards improving practitioners’ understanding of the SBCA process. Survey respondents also indicated a gradual decline in comprehension of the SBCA process when nearing Phase 3, potentially due to SI’s infancy.
Knowledge and Skills
As demonstrated in the following graph, practitioners and managers were most confident in the areas of teamwork, familiarization with sustainment concepts and critical thinking, and less confident in creativity/innovation, business acumen and data analytics.
Figure 6 Details - Survey results indicate that practitioners and managers were most confident in the areas of teamwork, familiarization with sustainment concepts and critical thinking, and less confident in creativity/innovation, business acumen and data analytics.
This figure is a horizontal bar graph depicts the six of categories assessed in this survey question: teamwork, sustainment concepts, critical thinking, creativity/innovation, business acumen, and data analytics). Results reflect the percentage of respondents who were in agreement that they were confident in their knowledge and skills in the respectful categories.
- 61% agree
- 54% agree
- 53% agree
- 44% agree
- 37% agree
- 33% agree
Some interviewees indicated that skills related to business acumen and innovation cannot be taught and must be acquired through experience, which may explain the lower levels of confidence exhibited by survey respondents in these areas.
21% of survey respondents that offered comments to improve the SI/SBCA process suggested increased training for practitioners; and
24% suggested simplifying/clarifying the SBCA guide.
When asked about practitioner knowledge and skills, interviewees indicated that skills were insufficient in areas such as Performance-based Contracting, business acumen and innovation. However, many interviewees found it challenging to comment on practitioner knowledge and skills due to limitations in resources, time and SBCA process flexibility which impact practitioner learning and development. In a 2018/2019 course evaluation report, 93% of respondents felt that the knowledge and skills presented during the Introduction to SBCA course would improve their work; however, only 27% felt they would be able to apply their learning into practice, suggesting a gap between theory and application. Addressing these shortcomings would contribute to the development of optimal sustainment solutions.
Collaboration and coordination efforts directed towards internal stakeholders have improved.
Finding 4: DND’s collaboration and coordination efforts with the CAF and its partner departments (PSPC and ISED) have improved, although some challenges remain. Industry collaboration is generally positive; however, practitioners and managers would benefit from further clarity on the Canadian defence industry’s role in the SBCA process.
Opinions are mixed among stakeholders regarding the effectiveness of the CAF’s input into the SBCA process. DGs and directors noted that CAF input has been beneficial in defining requirements and conducting options analysis; however, SIPM and CoE interviewees noted that the CAF does not always fully understand the purpose and objectives of SI. This presents an opportunity for further integration between DND/CAF stakeholders.
Overall, survey respondents are satisfied with DND/CAF and OGD collaboration, and are less satisfied with industry collaboration.
Figure 7 Details - Survey results indicate that overall, survey respondents are satisfied with DND-CAF and OGD collaboration, and are less satisfied with industry collaboration.
Collaboration between the three departments has improved, and is perceived as working well overall. Collaboration between PSPC and DND is functioning well at the director level; however, some challenges were noted at the working level, particularly conflicts regarding SBCA leadership between Contracting Authority and Technical Authority roles.
Collaboration with Industry
Practitioners and managers generally believe industry is being sufficiently engaged. Responses were slightly less positive with respect to the effectiveness of input being received from industry.
57% of survey respondents say industry engagement is at least somewhat sufficient; and
46% say input from industry is at least somewhat effective.
Similar to the industry collaboration results, a notable percentage of survey responses fell within the do not know or no response category (22% for industry engagement, 29% for industry input), which suggests a lack of understanding among practitioners and managers regarding industry’s role in the SBCA process. Additional key evidence related to this is discussed in slide 16. Finally, the majority of DND DGs and directors interviewed agreed that industry produced beneficial input/feedback. This suggests that engagement is more effective at the senior level than at the working level.
Practitioners’ and managers’ perception of the SBCA process value could be improved.
Finding 5: Efficiency-related concerns and communication challenges are hindering the extent to which practitioners and managers see value in the SBCA process.
Since 2016, efforts have been made to convey the initiative’s value through courses, workshops, announcements, materials and websites. Many interviewees noted that SI involves significant culture change, and that perceived value will increase as more SBCAs are fully completed and further benefits are identified.
Perception of SBCA process value
Evaluation evidence indicates that management generally sees value in the SBCA process, noting increased analytical rigor as an important benefit; however, reactions are more mixed at the working level. When practitioners and managers were asked to comment on which aspects of the SBCA process are/are not worthwhile, 18% noted positive aspects (e.g., allows DND to be a “smart consumer”); however, a number of efficiency-related concerns were raised, which are discussed in further detail in slide 14.
Communication challenges are also impacting the extent to which stakeholders see value in the SBCA process. Gaps in understanding of SBCA applicability have led some to question SBCA value, noting that SBCAs are “not relevant” for all sustainment solutions/contracts. Misconceptions regarding SI objectives were also identified, as a number of stakeholders viewed the primary goal of SI as cost savings versus optimization of all four sustainment principles. There also appeared to be a lack of awareness of recently achieved results, as many stakeholders were looking for more current, concrete results despite SIPM’s recent identification and communication of these.
Survey results suggest there is an opportunity to increase practitioners’ and managers’ perceived value of the SBCA process.
While 36% of responses were positive (“quite a bit” or “a great deal”), over half (54%) of responses fell within the “somewhat” or “very little” categories.
Figure 8 Details - Survey results suggest there is an opportunity to increase practitioners’ and managers’ perceived value of the SBCA process.
This figure is a vertical bar graph indicating a range of don’t know/no response, not at all, very little, somewhat, quite a bit, and a great deal. Results reflect the percentage of respondents who align with the statement in the figure.
Practitioners and managers perceived value of the SBCA process:
- Don’t know / No response: 4%
- Not at all: 7%
- Very Little: 25%
- Somewhat: 29%
- Quite a bit: 22%
- A great deal: 14%
Survey results suggest there is room for improvement regarding SI’s communication and engagement activities in this area. When asked to indicate the extent to which SI’s communication and engagement activities have contributed to increasing their recognition of the SBCA value, 45% of practitioners and managers indicated “very little” or “not at all” and 35% said “somewhat”.
Results regarding the CAF’s perspective on SBCA process value were mixed. The CAF representative interviewees generally perceive the SBCA process as valuable; however, other (DND management) interviewees noted that SBCAs are not a focus for the CAF, nor is their importance well understood.
Efficiency gaps remain in the SBCA process, and there is insufficient capacity to conduct SBCAs.
Since 2016, SI has demonstrated consistent efforts to enhance its efficiency. Examples include the development of a standardized process of engagement for the kick-off meeting in phase 1A of the SBCA process requiring all stakeholders to participate, streamlining the SBCA approval process, and the creation of a lessons learned repository to support practitioners working on SBCAs and development of best practices.
Overall, DND budget and actual expenditures for SI (DMPP-9 and CoEs) resources and personnel from FY 2016/17 to 2019/20 aligned*. Further, as noted on slide 16, individual roles and responsibilities in the SBCA process, as well as CoEs and OGDs, are generally understood.
Capacity and Resources
Gaps in CoE capacity were first identified in the 2017 Health Check, and SI management has worked towards addressing this.
CoEs have a considerable impact on SI and SBCAs; however, as noted by a former Sustainment DG in June 2019, SI needs to continue “leaning forward with staffing” and “cannot wait until SBCA work load exceeds capacity to find a solution.”
52% of SBCA supervisors reported that DND SBCA practitioners have insufficient human resources to conduct SBCAs.
About half of the practitioners (49%) as well as their supervisors (51%) reported that practitioners have insufficient time to conduct SBCAs, and when asked, process length and complexity were in their top three concerns regarding efficiency.
Training gaps (discussed earlier in slide 11) were also noted in survey and interview results (Sustainment Directors) as impacting efficiency. SIPM indicated that more guidance and tools for practitioners would benefit SBCA efficiency.
All sustainment and procurement directors interviewed emphasized capacity/resource limitations as the primary efficiency challenge. Sustainment DG interviewees also noted that the National Procurement budget is divided across environments and split between operations/maintenance and procurement, leaving little leeway.
SBCA Process Flexibility
All sustainment directors and several CAF interviewees indicated that the SBCA process would benefit from fewer and/or streamlined steps, increased clarity and reconsideration of timelines. However, SIPM/DMPP indicated that streamlining is already built into the SBCA process, and “people just don’t know it.”
34% of SBCA Practitioners and Managers find the SBCA process is too long.
24% find the SBCA process too complex.
*SIPM was unable to provide the information required during the course of the evaluation.
Gaps in SBCA accountability and inconsistent CoE delivery are hindering SI efficiency.
Finding 7: While several improvements have been made to SI’s governance structure and execution, inconsistent CoE delivery across environments is posing some efficiency challenges, along with gaps in SBCA accountability and centralized oversight.
Evidence suggests that inconsistencies in CoE execution has led to some confusion regarding the CoE role among stakeholders. Inconsistencies were identified regarding the CoEs’ approach to supporting SBCAs, with some taking a more prescriptive approach than others, as well as when SBCAs were being applied. The level of CoE support and CoE involvement also varies considerably, with the Air CoE having more capacity and taking a more involved role. This approach aligns with the 2019 CoE TOR, which calls for active involvement throughout the SBCA process, as well as feedback from practitioner and manager survey respondents who expressed a need for additional CoE guidance.
22% of interviewees and 19% of survey comment respondents noted inconsistent CoE delivery.
CoE interviewee responses were mixed with regards to the appropriate level of support for SBCA teams and whether there should be alignment between environments. While some CoEs advocated for active involvement, others supported a more distanced approach to encourage greater ownership of SBCAs.
Top themes across survey respondents that identified CoE delivery inconsistencies included:
Figure 9 Details - 80% of respondents indicated a need for more CoE support with regards to guidance and mentoring, and increased involvement with SBCA teams.
CoE current role, akin to a gatekeeper or challenge function, was perceived as impeding the efficiency of the SBCA process.
Figure 10 Details - 28% of respondents indicated that the CoE role is akin to gatekeeper or challenge function, and was perceived as impeding the efficiency of the SBCA process.
Since the 2017 SIHC, adjustments to the SI governance framework have been completed, a key one being updating the TORs for SIPM, SIMC and SISGC in 2018, and CoE in 2019. However, SIPM and CoE interviewees indicated a need for greater SBCA accountability at the DG level and higher to ensure consistency in SBCA execution, direction and messaging. DND sustainment DG interviewees also noted the need for more rigorous oversight from leadership. Until recently, there was no central oversight in place to ensure accountability for and to monitor the status of SBCAs and timelines across EPMs and MPDs.
11% of survey respondents who commented on suggestions for improvement noted leadership accountability as an area for potential improvement.
The new SI Champion position fulfilled by the Associate ADM(Mat) has recently provided central oversight to SI, encouraging DG reporting on SBCAs and providing consistent direction to the major projects and equipment programs, particularly with regards to SBCA applicability.
There is room for improvement with respect to the roles and responsibilities of the Canadian defence industry.
Finding 8: Practitioners and their managers generally understand their individual roles and responsibilities in the SBCA process, as well as DND’s (incl. CoEs), the CAF representatives’ and Other Government Departments’; however, the Canadian defence industry’s roles and responsibilities are less clear.
Individual roles and responsibilities
Survey results indicate that DND SBCA practitioners and managers sufficiently understand their roles and responsibilities related to conducting or supporting the SBCA process.
The majority of surveyed practitioners and managers have a good or very good understanding of their roles and responsibilities.
Figure 11 Details - The majority of surveyed practitioners and managers have a good or very good understanding of their roles and responsibilities.
Roles and responsibilities of partners
Survey results indicate that understanding of the roles and responsibilities of partners involved in the SBCA process is good overall. Canadian defence industry’s roles and responsibilities were less understood.
Figure 12 Details - Roles and responsibilities of partners involved in the SBCA process.
SI’s draft PMF needs to be finalized and implemented to further assess impact.
Finding 9: SI recently developed a draft PMF, which needs to be refined and finalized.
Performance Measurement Progress
Interview and documentary evidence indicates that SIPM has had some challenges in developing appropriate performance measurement metrics. Interviewees noted difficulties in commenting on the quality of SBCA solutions due to the lack of an implemented PMF; however, they also acknowledged the challenge in doing so as each sustainment solution is different.
SIPM recently established a draft PMF that includes SI and SBCA-level metrics, which was approved by SISGC in 2019 with some feedback to be addressed. SIPM also provides some guidance on performance measurement in the options analysis phase of the SBCA Guide and indicates that performance strategies are to be implemented at the sustainment solution monitoring phase.
T56+F404 Engine SBCA Example
- Close to 100% performance targets being met
- Payments based on completion rather than progress
- Flexibility integrated to adapt to changing RCAF requirements
3 out of 5
DND sustainment directors noted the absence of common metrics as a challenge to measuring the success of the SI.
Planned Performance Measurement
SIPM’s draft PMF is well-positioned as it assesses the quality of SBCA process itself (i.e., ensures operational sustainment requirements are sufficiently defined) – which is the foundation for developing optimal sustainment solutions. It also ensures a well-defined PMF structure will be put in place that can be supported by practical data sources.
The PMF also supports the assessment of SBCA “impact” by assessing performance against the four principles, with specific performance indicators to be developed and collected by SBCA teams and reported annually. With SI having been implemented for four years, it is important that performance information be collected systematically to support the ongoing assessment of SBCA/SI performance, as well as to support decision making and program improvements.
Finally, the PMF’s SI-level metrics are useful as “health indicators”; however, they are limited in the extent to which they can be attributed to SI. For example, several factors external to SI may contribute to ADM(Mat)’s “fleet availability,” which is the current performance measurement for SI’s “performance” principle.
DND has made progress towards achieving its intended SI outcomes since 2016. Key accomplishments include increased practitioners’ and managers’ understanding of the SBCA process and sustainment principles; improved collaboration and coordination with key stakeholders; and early indications of optimized sustainment principles. However, efficiency-related issues - particularly the lack of CoE and practitioner resources to conduct SBCAs - are hindering DND’s success. Gaps in communication of SBCA process value, practitioner training and support, and clarity of the Canadian defence industry’s role are also limiting SI’s ability to achieve its outcomes. Finally, SI’s PMF needs to be finalized and implemented to measure SI’s success in the near future.
There is a continued need for the SI.
- Defence sustainment accounts for a considerable amount of the Defence budget, and sustainment costs are expected to increase due to aging fleets and increasingly complex equipment.
- The increasing trend in SBCA workload also suggests an ongoing need for sustainment.
- SI supports CAF operational readiness by ensuring members are well- equipped with optimal sustainment solutions, ensures accountability for resources, and benefits Canadians writ large by encouraging job creation and economic growth.
DND has made progress in achieving its immediate outcomes of increasing sustainment practitioner competencies, collaboration and coordination with key stakeholders and stakeholder perception of SBCA process value; however, more work is needed in the following areas:
- Sustainment practitioners require further support and training to further develop the necessary competencies required for developing optimal sustainment solutions;
- Further clarification is needed with regard to the Canadian defence industry’s engagement in the SBCA process; and
- Efficiency-related concerns and communication challenges identified as hindering the extent to which practitioners and managers see value in the SBCA process need to be addressed.
SI has contributed to its intermediate outcome of developing informed sustainment solutions and optimized principles.
- Through the SBCA process and its Options Analysis phase, SI contributes to informed sustainment solutions that optimize the four sustainment principles.
- Practitioner and manager survey results were positive in this regard, and examples were identified demonstrating optimization of sustainment principles.
- It will be important to continue to collect such examples to measure SI success, as well as to further demonstrate the value of the SBCA process to stakeholders moving forward, as SI matures and more SBCAs are fully completed.
Despite some indications of SI efficiency, important gaps remain.
- CoEs and practitioners lack capacity, resources and flexibility to conduct SBCAs.
- Inconsistent CoE delivery across environments is also posing challenges, as are gaps in SBCA accountability and centralized oversight.
- There is also lack of clarity among DND practitioners and managers in Canadian defence industry’s role and responsibilities in the SBCA process.
- Finally, SI recently developed a PMF; however, this needs to be finalized and implemented in order to measure SI success in the future.
- Sustainment practitioners are expected to have a base level of competency before they begin to develop an SBCA. Base competencies, while critical to developing a sustainment solution, are outside the scope of the SI.
- Therefore, by March 31, 2021, the Materiel Group will map out the SBCA competencies required by SBCA practitioners, including competencies identified by the evaluation team.
- By March 31, 2022, the Materiel Group will develop a plan to address the SBCA-specific competency gaps. Content development and delivery funds will be requested starting in FY 2022/2023.
Target Date: March 31, 2022
- As part of the Sustainment Institutionalization Plan, the role of industry and the further engagement of the CAF will be explored.
Target Date: November 2022
- By January 1, 2021, the Materiel Group will hire a manager focused on communication during a change initiative.
- By September 30, 2021, the Materiel Group will have amended its Sustainment Initiative Communication Plan to focus on change management. This will include emphasizing the value of the SBCA process.
Target Date: September 2021
- By December 30, 2021, the Materiel Group will test a new SBCA manual using a case study methodology. One of the focuses of the test will be to ascertain areas for efficiency. The new manual will be divided between mandatory SBCA components and optional or best practices.
- By March 31, 2022, the Materiel Group will publish the new SBCA manual, including any identified process efficiencies.
Alignment of resources with workload
- Under the Associate ADM(Mat), the Sustainment Solution accountability of DGs has been rejuvenated. As of October 20, 2020, DGs report on the status of their sustainment solutions and the health of their sustainment programs. Divisional resources are allocated commensurate with competing divisional priorities.
- By March 31, 2022, the Materiel Group will publish the new SBCA manual. This will include the expected quality of SBCA reports and SBCA solutions so that divisions will be able to make resource return on investment decisions.
Target Date: March 31, 2022
- Under the associate ADM(Mat) the sustainment solution accountability of DGs has been rejuvenated. As of October 20, 2020, DGs report on the status of their sustainment solutions and the health of their sustainment programs. By March 31, 2021, DND’s oversight on SBCAs will be formalized in an appropriate policy instrument.
- By March 31, 2022, the Materiel Group will publish the new SBCA manual. This will provide clarity to the CoEs on the process as well as the expected quality of the SBCA reports.
Target Date: March 31, 2022
- A manager has been given the primary responsibility to develop a PMF. The initial framework will be developed by March 31 2022 and will consist of three phases. 1. The assessment of the state or status of the CoEs and sustainment practitioners; 2. The assessment of the Sustainment Solutions Reports; and 3. The assessment of the sustainment solution over time.
- By March 31, 2022, the Materiel Group will publish the new SBCA manual. This will include the expected quality of SBCA reports and SBCA solutions.
Target Date: March 31, 2022
Coverage and Responsibilities
Due to the SI’s recent implementation (officially launched in 2016), the evaluation covers the four-year period from FY 2016/17 to FY 2019/20.
The evaluation examines the relevance (i.e., continued need), effectiveness and efficiency of the SI’s activities. This evaluation concentrates on activities specific to DND, as PSPC’s and ISED’s activities were considered outside of scope. However, coordination and collaboration between the three departments was assessed. Additional information on evaluation approach, methodology and limitations can be found in Annex D.
Between March and June 2017, an SIHC was conducted by ADM(Mat) staff with support from ADM(RS). Its objective was to assess the progress made in implementing SI principles and process across all three departments. As SI was still in its infancy, the SIHC assessed the extent to which the foundational aspects were in place to enable future and enduring success. Where applicable, the evaluation used the SIHC report as a baseline, to assess SI’s progress towards achieving outcomes since 2017.
In January 2018, a Joint Statement to Practitioners noted that SI had completed the phase-in period, which consisted of a number of pilot projects and 20 SBCAs. That year, the examination of SI’s benefits for DND/CAF was requested by the 2018 ADM(Mat).
Key SI activities include the following:
1. SBCA Conduct and Support
DND sustainment practitioners conduct SBCAs with guidance, coaching and mentoring provided by interdepartmental CoEs.
SI offers the following primary support resources: an SBCA Guide explaining the process; a GCpedia webpage explaining the initiative; an online introduction to SI course; a three-day in-person introduction to SBCA workshop; an Executive workshop; and a CoE tri-departmental team for each environment.
2. Policy and Procedures
Conduct continuous policy and process improvements (e.g., SBCA Guide and support procedural document revisions and enhancements).
3. Resourcing and Professional Development
Assign resources to conduct SBCAs; provide professional development/training opportunities (courses, workshops)
4. Communication and Engagement
Develop and implement communication and engagement products/strategies - internally within DND as well as with external stakeholders (e.g., industry).
Develop and maintain SI leadership and oversight within DND (SI Program Management) and across the three departments (i.e., DG-level SIMC, ADM-level SISGC).
Activities related to sustainment include, but are not limited to: maintenance, repair and overhaul; engineering support; engineering test and evaluation; maintenance program management; fleet management activities; acquisition of parts; logistics management; and sustainment-related training.
Due to the initiative being in the early stages of implementation, the evaluation focused on the assessment of short-term outcomes, and developed forward-looking indicators to inform on SI’s current and planned development and implementation.
- Increased competencies to develop tailored sustainment solutions;
- Improved collaboration and coordination among stakeholders; and
- Increased recognition of the value of the SBCA among stakeholders.
- Development of informed, tailored sustainment solutions that optimize performance, value for money, flexibility and economic benefits.
Data Collection Methods
The evaluation used multiple lines of evidence and complementary qualitative and quantitative research methods (described in more detail in the following slide) to help ensure the reliability of information and data supporting findings:
- key informant interviews;
- online survey;
- document review;
- financial/Human Resources (HR) data analysis; and
- direct observation.
Data Collection Methods
The lines of evidence collected through the conduct phase of the evaluation were triangulated with each other and verified with program officials to ensure their validity. The research methodology used in the scoping and conduct of the evaluations are as follows:
Key Informant Interviews
The evaluation team worked with SIPM (and CoE leads in some cases) to identify interviewees and receive feedback on the interview guide. There were a total of 31 interviews conducted with the key stakeholder groups outlined in the following table. The evaluation team worked with CoE leads to ensure sufficient representation was met for the three environments, and selected interviewees who had the most experience conducting SBCAs, and who had advanced the furthest in the process (i.e., phase 3).
|Internal to DND/CAF:|
CoE (1 interview/environment)
DND Sustainment Directors
DND Sustainment DGs
DND Procurement Directors
DND Associate ADM/SI Champion
External to DND/CAF:
OGDs (PSPC and ISED)
Table D-1 Details - Key Informant Interviews (31 meetings total)
An online survey was developed for DND/CAF sustainment practitioners and managers/supervisors who are involved in SI. It was designed in consultation with SIPM and CoE leads. The survey focused on assessing the initiative’s performance (effectiveness and efficiency). It was launched over a two-week period (June 1– June 12, 2020), and distributed using a Chain of Command approach, initiated from the Associate ADM (SI Champion) down to the working level. Overall, there was a response rate of ~ 50% (208/~400); however, only 160 respondents indicated their involvement in conducting/support SBCAs, which was a key condition of the evaluation. Thus, only responses from the noted 160 participants were used in the analysis phase of the evaluation. Indications from SIPM suggest that adequate representation was met across the five applicable groups (Director General Land Equipment Program Management, Director General Maritime Equipment Program Management, Director General Aerospace Equipment Program Management, Director General Major Project Delivery (Air and Land), Director General Major Project Delivery (Sea)).
A preliminary review of foundational documents was conducted during the planning phase of the evaluation to ascertain a fulsome understanding of the program to inform the development of the scope, logic model and evaluation matrix. An official Request for Information was then submitted to SIPM to obtain all pertinent documentation that would assist in the program assessment. The evaluation team received over 100 documents, which included: government websites; departmental administrative reports; program documents, some draft, others final; external reports; and policies and mandates applicable to the program.
Financial/HR Data Analysis
A review of financial and HR data was conducted to assess the efficiency of the program since its official launch in 2016. Information was obtained from the three CoEs, DMPP and SIPM.
A member of the evaluation team attended an executive SBCA workshop led by the program to observe the method of teaching, and level of engagement/feedback of participants. No other opportunities presented themselves due to limitations resulting from the pandemic.
The limitations encountered by the evaluation and mitigation strategies employed in the evaluation process are outlined in the following table.
SI is a tri-departmental initiative, thus, a DND-
focused evaluation may not capture all aspects of the program’s relevance, effectiveness and efficiency.
|The evaluation team focused on the achievement of outcomes prioritized by DND, specifically, and has put forward recommendations that reflect DND’s scope of influence.|
The initiative is in the early stages of implementation. Therefore, the evaluation’s scope was limited to assessing short-term
To the extent possible, the evaluation team incorporated forward-looking indicators to inform on SI’s current and planned development and implementation.
Limited performance data/information available (e.g., PMF has not been implemented).
The evaluation relied more heavily on the following lines of evidence to inform the findings and recommendations: key informant interviews; a survey; financial/HR analysis; direct observation; and document review.
|Qualitative vs. Quantitative||
Due to the nature of the outcomes on which the evaluation focused, there is a potential risk that the evidence obtained may be largely perception-based.
This limitation was addressed by triangulating evidence from multiple data sources to inform the findings and ensuing recommendations (e.g., interviews from stakeholders at varying levels and with a range of experience across DND/CAF, review of pertinent program documentation, financial/HR data analysis, direct observation of program participants, and a survey to assess the experiences of working-level staff).
Unforeseen change in working environment due to a global pandemic could inhibit the evaluation team’s ability to complete the project on time.
Evaluation team members were able to make use of collaborative tools to move forward with the conduct, analysis and report writing phases of the project, while working remotely.
Table D-2 Details - Evaluation Limitations.
Report a problem or mistake on this page
- Date modified: