Evaluation of Modernization of Hosting Services (Linux/Unix)

Permission to Reproduce

Except as otherwise specifically noted, the information in this publication may be reproduced, in part or in whole and by any means, without charge or further permission from Shared Services Canada, provided that due diligence is exercised to ensure the accuracy of the information reproduced is maintained; that the complete title of the publication is produced; that Shared Services Canada is identified as the source institution; and that the reproduction is not represented as an official version of the information reproduced, nor as having been made in affiliation with, or with the endorsement of the Government of Canada.  

Commercial reproduction and distribution are prohibited except with written permission from Shared Services Canada. For more information, please contact Shared Services Canada at publication-publication@ssc-spc.gc.ca. 

© His Majesty the King in Right of Canada, as represented by the Minister responsible for Shared Services Canada, 2025. 

Evaluation of Modernization of Hosting Services (Linux/Unix)
Cat. No. P118-37/2025E-PDF 
ISBN  978-0-660-78197-6 

Publié aussi en français sous le titre : 
Évaluation de la modernisation des services d’hébergement (Linux/Unix) 
No. de catalogue P118-37/2025F-PDF 
ISBN  978-0-660-78198-3 

Executive summary

The Office of Audit and Evaluation conducted an evaluation of the modernization of hosting services (Linux/Unix). The evaluation assessed responsiveness, effectiveness and efficiency. This report represents a snapshot in time from 2020-21 to 2023-24, and reflects data collected in 2024. Linux/Unix services were identified as the focus of the evaluation as these services represented both the opportunities and the challenges associated with providing hosting services. This evaluation is intended to inform decision making and identify potential implications for future operations.

Key findings

Scope and completion target

In 2020, Shared Services Canada (SSC) launched the Operating System Modernization Initiative (OSMI) to modernize over 9,000 Linux/Unix operating systems for 31 partner organizations. The scope of this initiative expanded by 22% over time as incomplete asset records were corrected. As of October 2024, SSC had completed 62% of the OSMI modernization.

Although OSMI was not on track to meet its original target set for March 2026, substantial progress had been made. From a technical perspective, progress slowed over time because the operating systems became generally more difficult to upgrade or decommission, having focused on the “quick hits” early on. In addition, technology evolved over time, thus operating systems that were not originally considered out-of-date required upgrades to remain current with new technology. Despite these challenges, it is expected that approximately 88% of the modernization objective will be completed by March 2026.

Impact on business value

From a Government of Canada (GC) perspective, the impact of unmet modernization target on business value is unclear. It is not known how this might affect partner services to Canadians, and how this might impact the desired stewardship outcomes and results that SSC provides for the GC. Presumably, partner departments would have pressures to support the modernization of the operating systems for their critical and other important business applications and systems which their departments require to deliver their mandates. This suggests that the remaining modernization tasks may be of relatively lower value. From an enterprise perspective, legacy compared to modern systems present increased vendor support costs, higher security risks, greater complexity, and constraints to introducing automation and innovations.

SSC Linux/Unix staff

A key factor that supported the modernization of Linux/Unix hosting services was the dedication and professionalism of the Linux/Unix staff. Partners independently and repeatedly spoke of their positive experiences with the Linux/Unix staff, contributing to increased confidence among partners in SSC as a service provider and the department’s reputation for quality.

Incentives

At the same time, a number of factors hindered modernization. Some of these hindering factors were specific to Linux/Unix hosting services. They included a lack of mechanisms to incentivize partners to modernize, a lack of clarity among partners on SSC’s modernization strategies, and deficiencies in the department’s asset and configuration management system:

• Incentives: During the evaluation period, partners lacked motivation to transition to modern Linux/Unix operating systems in a timely fashion. At the same time, SSC did not offer sufficient incentives to encourage partners to prioritize and actively participate in the modernization process.
• Clarity of SSC plans: Both partners and SSC employees reported being unclear about SSC’s strategic and operational plans for the modernization of hosting services. Partners reported that they struggled to understand SSC’s priorities and timelines for the modernization of hosting services. They reported that the absence of concrete, detailed implementation plans made it challenging for them to align their plans with SSC’s modernization efforts.
• Information gaps: Deficiencies in SSC’s collection of asset and configuration management system impacted modernization progress. Specifically, SSC lacked effective tools for asset tracking and configuration management. This hampered SSC’s ability to meet modernization objectives and timelines. It also directly impacted the security and reliability of the GC’s IT infrastructure and services.

Other challenges

More broadly, there were also department-wide challenges that hindered progress towards modernization. These included inefficiencies in the intake processes, the quality of costing and financial information for decision making, talent acquisition and retention challenges, and weak performance measurement for outcomes:

• Inefficient intake processes: SSC’s intake processes and related tools were cumbersome and this affected the Linux/Unix Division’s ability to deliver services in a timely and efficient manner. The deficiencies in the intake processes were further compounded by a lack of internal coordination, which had a negative impact on SSC’s capability to properly manage workload and allocate resources.
• Ineffective financial tools: SSC’s existing financial tools and related processes hindered SSC’s ability to provide accurate and reliable cost estimates. This created disincentives to modernization for partners and negatively impacted SSC’s reputation as a service provider. In the absence of an integrated financial management and cost accounting system, SSC has relied on budget allocations at the branch level to control costs.
• Staffing challenges: Talent acquisition and retention challenges for Linux/Unix expertise affected SSC’s ability to achieve operational goals and deliver modernization solutions. The competitive marketplace for specialized talent, rigid staffing processes, and language requirements contributed to unfilled positions within the Linux/Unix Division. As of November 2024, 21% of the established positions in the Linux/Unix Division were vacant.
• Weak performance measurement: Lastly, the foundational performance measurement framework for the “Data Centre IT Operations” program was inadequate in measuring outcomes and the value provided by SSC. Key deliverables did not adequately capture the impact of either excellent or poor performance of the Linux/Unix program component for partners or track the benefits to the GC from modernization and improved stewardship.

Figure 1 presents an overview of the various factors that either supported or hindered modernization.

Figure 1. Factors that supported and hindered modernization

Recommendations

The key findings of the evaluation led to the following recommendations:

Recommendation #1 (HSB): In collaboration with TBS, establish incentive mechanisms to encourage partners to support modernization objectives and comply with timelines.

Recommendation #2 (HSB): Refine and communicate detailed strategic and operational plans for hosting services. This could include:

• Processes for consultation and collaboration with partners;
• Definition of clear objectives and realistic timelines;
• Information on resource requirements for each modernization phase, as this would aid partners in planning their work;
• Specifying feedback channels to allow for clarifications and adjustments.

Recommendation #3 (HSB): Improve the effectiveness of SSC’s IT Asset Management and Configuration Management System. This could include:

• Updating asset and configuration management databases to ensure reliable and up-to-date asset and configuration information;
• Standardizing asset and configuration management processes to ensure consistency and reliability of asset and configuration data;
• Leveraging automated tools and processes to reduce human error and provide real-time updates.

Recommendation #4 (OCSB/HSB): Review and streamline elements of the intake processes to identify and address inefficiencies. This could include:

• Defining, documenting and communicating the processes required for delivering Linux/Unix hosting services through either a business request, service request, or change request;
• Ensuring all essential requirements are collected at intake for decision-making.
• Updating the service catalogue to improve accuracy, relevance, and accessibility of information for partners;
• Improving integration between service management tools (where possible) to enhance visibility of ongoing requests for partners and SSC service lines;
• Streamlining the ordering process for services for partners and update ONYX.

Recommendation #5 (CFOPB): Improve the quality and integrity of cost information within the existing pricing tool to ensure that SSC receives sufficient funds to cover expenses and costs incurred to deliver BRs. This could include:

• Sampling BRs after completion and verifying costs to validate information within the existing tool and improve pricing decisions;
• Improving ePET training materials to reduce user errors. This could include communicating roles and responsibilities for accurate data entry.

Recommendation #6 (HSB): Enhance the recruitment and retention of Linux/Unix expertise through more flexible recruitment processes. This could include:

• Establishing specialized staffing processes for Linux/Unix technical positions on a more frequent basis;
• Reviewing the language requirements for Linux/Unix technical positions to ensure alignment with the position requirements and consider flexibility where appropriate measures are put in place.

Recommendation #7 (HSB): Update the Data Centre IT Operations Performance Information Profile to include meaningful outcome measures and appropriate performance indicators, including baselines and target levels of improvement. The logic model should include service delivery outcomes for partners and stewardship outcomes for the GC.

1. Introduction

This evaluation assessed the responsiveness, effectiveness and efficiency of the modernization of SSC’s hosting services, focusing on Linux/Unix services. It is intended to inform decision making and identify potential implications for future operations. The evaluation was conducted by the Office of Audit and Evaluation. It represents a snapshot in time based on data collected in 2024 and covers the period from 2020–21 to 2023–24.

2. Context and progress of modernization

This section outlines the context facing Linux/Unix Hosting Services, the Operating System Modernization Initiative, and the progress made in Linux/Unix modernization.

2.1 Linux/Unix hosting services context

During the evaluation period, thousands of software applications were used to run the systems that delivered Government of Canada (GC) programs and services to Canadians. Many IT systems and infrastructure components hosting these applications were complex, non-standardized, and outdated, making them costly to maintain and subject to higher security and reliability risks than modern systems.

In 2023, SSC launched Delivering Digital Solutions Together for Canada (Digital Together). The goal of the Digital Together strategy was to ensure that government departments were equipped with modern and standardized tools to effectively collaborate and better serve Canadians. Digital Together focused on 4 key SSC service areas: connectivity services, hosting services, digital services, and cyber security services.

As part of Digital Together, the vision for the evolution of hosting services resulted in the creation of the new Hosting Services Branch (HSB). This branch brought together the totality of SSC’s hosting solutions to better position the department to deliver on partner needs through an enterprise approach. It aligned with the GC direction to go “cloud smart,” modernize applications, reduce technical debt, and ensure the continuous optimization of service delivery. HSB was responsible for delivering the Data Centre IT Operations program, which was composed of a number of distinct program components including Linux/Unix management.

The Linux/Unix management program component was delivered by the Linux/Unix Division. The objective of the division was to enable all GC partner organizations to benefit from modern, standardized, consolidated, responsive and reliable hosting services. During the evaluation period, the division managed approximately 10,000 Linux/Unix-based servers supporting thousands of applications, many critical to the operations of partner departments that provided services to the GC or directly to Canadians.

Additional information on the roles and responsibilities for the Treasury Board of Canada Secretariat (TBS), SSC and partner departments can be found in Appendix A.

2.2 The Operating System Modernization Initiative

Due to decades of underinvestment, IT infrastructure within the GC has aged faster than the pace of repairs or replacement. As of April 2020, out of approximately 9,000 known Linux/Unix operating systems, 5,047 were either soon to be or no longer supported by vendors.^{Footnote 1} This made it difficult for partner organizations to resolve system-related operational issues, leading to potential downtime and a loss of productivity. As vendors stopped providing security updates and patches, these systems also became more vulnerable to cyber threats.

In 2020, SSC launched the Operating System Modernization Initiative (OSMI) to modernize and standardize Linux/Unix operating systems for 31 partner organizations. The OSMI was expected to yield significant improvements across several areas including: security, reliability, resource optimization, cost savings, and the modernization of legacy IT systems. The initiative was aligned with the GC’s larger IT strategy and digital transformation agenda.

The objectives of the OSMI included:

• Identifying and upgrading obsolete and unsupported infrastructure hosted in legacy environments;
• Reducing the types of Linux/Unix operating systems;
• Aligning operating systems to enterprise standards; and
• Preparing partner environments for migration to enterprise data centres or the cloud.

The OSMI team was moved under the Linux/Unix Division on May 1, 2024.

2.3. Linux/Unix modernization progress

At the outset, the OSMI aimed to modernize over 9,000 Linux/Unix operating systems for 31 partner organizations by 2025–26. It was later found that due to incomplete records, not all operating systems that SSC supported were included in the initial inventory. This meant that as of October 2024, the estimated total number of Linux/Unix operating systems in scope had increased by 22% (from 9,222 to 11,242). Based on this updated inventory, SSC has migrated or decommissioned a total of 6,946 of these reported operating systems as of October 2024, representing 62% of SSC’s modernization objective.

Source: OSMI Dashboard

Despite challenges, the OSMI is expected to attain 88% of its modernization target by the end of fiscal year 2025-26. Figure 1 shows that modernization progress slowed as complexity increased, and technology evolved.

Additionally, much of the early modernization work focused on operating systems that were relatively easy to upgrade or decommission. This explains why the average pace of modernization in the first few years (e.g., April 2022 to March 2023) has been declining. This slowdown was expected given that the OSMI would be tackling more complex operating systems, requiring extensive customizations that were difficult to replicate or transfer to new environments.

With time, the number of systems in need of modernization could also increase due to changes in technology standards. As technology has continued to evolve, instances of operating systems that had not previously been considered out-of-date could become candidates for modernization.

In terms of business value, it is not clear what the impact of missing this objective will be either for the services delivered by partners to Canadians or for desired stewardship outcomes and results for the GC. Presumably, partner departments would likely have pressures to support the modernization of the operating systems for their critical and other important business applications and systems which their departments require to deliver their mandates. This suggests that the remaining modernization tasks may still be important but are likely to be of lower priority. For the GC, the persistence of legacy operating systems results in increased vendor support costs, higher security risks, and greater complexity than necessary for SSC to manage, as compared to modern systems. It also limits flexibility for introducing automation and other innovations from the private sector.

3. Evaluation findings

Overall, this evaluation found that there is a genuine appreciation by partners towards SSC staff, in spite of the challenges that hindered modernization and efficiency, including the absence of incentives to encourage partners to step up and support modernization, insufficient direction and guidance from SSC to assist partner planning, and asset and configuration management information deficiencies.

More broadly, there were departmental challenges that related to inefficiencies in the business request process, the quality of costing and financial information for decision-making, talent acquisition and retention challenges, and weak performance measurement for outcomes. Figure 2 presents an overview of the various factors that either supported or hindered modernization.

Figure 3. Factors that supported and hindered modernization

3.1 Partners appreciated the dedication and professionalism of SSC staff

Partners repeatedly and independently reported positive experiences working with the staff members of the Linux/Unix Division. Specifically, they acknowledged the Linux/Unix Division as a dedicated team of hardworking professionals. In interviews, partners commented positively on the outstanding support and customer-oriented attitude. Many partners reported that the operational staff at the Linux/Unix Division went above and beyond to address their needs and concerns. Partners’ positive experience with SSC staff contributed to increased customer satisfaction, and in turn to increased confidence in SSC as a service provider and SSC’s reputation for quality.

Figure 4 – Quotes from partners’ positive experiences with SSC’s Linux/Unix staff

3.2 SSC lacked mechanisms to incentivize partners to modernize

Modernizing and standardizing Linux/Unix operating systems required SSC to work cooperatively with partner organizations to access partner IT infrastructure and environments. Partners also needed to upgrade and rebuild their older information systems and applications to be compatible with the latest Linux/Unix platforms. There were several factors that led to low partner motivation for this cooperation and modernization. First, the financial incentives were lacking since partners would continue to pay the same service fees to SSC after transitioning to more cost-efficient, modern solutions.

“From an incentive perspective, it [upgrading and modernizing] doesn’t give them [partners] anything. It doesn’t cost them less, and it doesn’t give them more security. So for them to have to pay for an infrastructure, put it in place in their data centre and pay for the lease cost etc., is fairly prohibitive.”

Second, while maintaining and protecting legacy systems typically required extended vendor support, the cost for this was borne by SSC not the partners. As a result, there were no direct negative cost implications for partners who remained on legacy systems.

Third, the increased security risk of these older systems was not viewed as a disincentive for partners who did not modernize, since SSC was responsible for preventing and mitigating the security-related risks on a legacy Linux/Unix service.

Finally, many partners were content with existing systems since newer systems implied additional training requirements, and partners generally believed that adopting newer operating systems required additional financial and human resources overall.

Both the partner interviews and the survey revealed that partners (49%) and SSC employees did not believe SSC was providing sufficient incentives to encourage support of modernization objectives, including timelines. These findings are in line with the literature review conducted during the evaluation.

A lack of incentives for modernizing is a central challenge that prevents many public-sector organizations from implementing successful infrastructure modernizations.

SSC employees also believed that simply relying on the goodwill of partner departments was insufficient to encourage modernization, and stressed the need to develop effective incentive mechanisms to encourage partners to prioritize and actively participate in modernization. Interviewees suggested that this might require the direct involvement and support from TBS, which, by providing policy direction and oversight, could help ensure that any newly developed incentive mechanisms would be effective and aligned with the GC’s modernization efforts.

3.3 Lack of clarity among partners of SSC’s strategic and operational plans made it difficult for partners to align with SSC’s modernization efforts

At the time of data collection in summer 2024, partners expressed uncertainty regarding SSC’s strategic plan and priorities when it came to the modernization of hosting services. The lack of concrete details made it challenging for partners to align their goals with SSC’s modernization efforts. In interviews, some partners reported that they did not fully understand the objectives of the OSMI and how it could benefit their organization directly. In the July 2024 survey, only 31% of partners agreed that SSC’s communications relating to its strategic direction for Linux/Unix hosting services were clear.

On the strategic level, we are not sure what direction SSC is going and how it will impact us.

This view was shared by SSC employees. Interviewees confirmed that even Linux/Unix staff members were unclear about SSC’s detailed strategic plan for hosting services. While SSC’s Digital Together strategy and the Hosting Services Roadmap contained high-level vision statements for a reliable and sustainable hosting ecosystem, there was no specific guidance for effective implementation. As a result, Linux/Unix employees reported that they struggled to explain to partners how SSC’s strategy would impact their day-to-day operations.

I’m not sure that the GC is aware of the way we’re going to evolve beyond the fact that we have said really nice words about what our intentions are.

The absence of concrete planning and implementation details further affected partners’ ability to meet modernization objectives and SSC schedules. Partners reported that they did not have sufficient information about the timing and sequencing of specific modernization steps for operational planning. They shared numerous examples of last-minute requests from SSC in preparation for an upcoming migration or upgrade. These requests ranged from installing anti-virus software to updating applications. As partners were unprepared for these requests, they had limited time to implement and test the required installations. This led to missed deadlines and delayed modernization progress.

Sometimes we were told [to migrate], the purpose is not clear and the timeline is not realistic. Often we cannot meet [the timelines].

The lack of clarity about planning and implementation details also impacted the effectiveness of service delivery. Interviewees stated that, due to the lack of foresight, there was limited ability for the Linux/Unix Division to align resource capacities with projected changes. Consequently, the division was constantly responding to partner demands as they arose. While services may have been delivered in a tactical manner, there was a lack of proactive planning to help anticipate future needs and optimize IT resources.

In order to create a coherent approach for the GC’s modernization objectives, there was a need for a clear strategic plan and effective communication of this plan. Partners requested more information on SSC’s modernization priorities and timelines to foster better coordination and alignment between SSC and partner departments.

3.4 Deficiencies in asset and configuration management systems impacted modernization progress

During the evaluation period, SSC was responsible for managing a wide variety of government IT infrastructure assets for which it required accurate configuration management information.

3.4.1 Asset management

In order to manage IT infrastructure assets effectively and efficiently, SSC needed to accurately track and monitor them. A robust asset management system for IT infrastructure assets would ensure efficient tracking, management, and optimization of IT infrastructure assets throughout their entire lifecycle.

During the evaluation period, SSC’s asset management system was comprised of a suite of tools and procedures. Using this system, asset tracking required manual data entry into laborious filing systems, which was time consuming, inefficient, and prone to errors. The errors were further compounded when data was transferred across multiple filing systems. Linux/Unix employees reported that approximately 30 spreadsheets were involved in various stages of the asset tracking process. Moreover, there was a lack of integration mechanisms for managing their interactions. This resulted in a segmented approach with scattered information across the department.

SSC does not have a good asset inventory for my organization. We are often asked to validate their information, or to gather it in the first place.

The deficiencies in the asset management system also affected partner confidence in SSC’s service delivery. The initial stage of an OSMI project involved an inventory review with the partners to identify the scope of work. This process often revealed incomplete or obsolete partner inventories. This raised concerns among partners regarding SSC’s asset tracking and monitoring capabilities. In the survey of partners, only 32% of respondents believed that SSC’s asset management for Linux/Unix hosting services was effective.

3.4.2 Configuration management

Configuration management was one of the five core processes for IT Service Management (ITSM). To successfully modernize operating systems across partner departments during the evaluation period, SSC needed accurate configuration information such as application dependencies, hardware specifications, networking details, and application-to-server mapping. A robust configuration management system would include a configuration management database (CMDB) and integrated processes. Each time there was a configuration change, detailed information about the change would need to be recorded in the CMDB.

“Due to application compatibility issues, [standardization and OS modernization] often go unrealized even after two years.”

During the evaluation period, partners indicated that SSC did not have valid information about the compatibility between partner applications and operating systems. This hindered the ability of Linux/Unix teams to meet modernization objectives and timelines. For example, at the time of a scheduled migration, an identified incompatibility could put migration on hold until the incompatible application was upgraded. In many cases, the delay was extended due to reasons such as partner resistance or resource constraints. During interviews, both SSC employees and partners shared examples of missed modernization timelines, ranging from several months to years.

“We were making a change for one of SSC’s products. I find out this morning there was a 7-hour outage as we were given improper configuration.”

Also, during the evaluation period, SSC did not have sufficient information on application-to-server relationships. One of the objectives of the Enterprise Control Desk (ECD) was to ensure an inventory of critical applications; however, only about a third of applications on the ECD list were mapped to servers.

Without proper information on application-to-server relationships, SSC did not know which GC applications could be impacted in the event of a server crash or a security vulnerability. This could have resulted in significant impacts on Canadians who relied on the services provided by departments and agencies dependent upon these applications.

“There are usually several ‘orphan’ servers that cannot be mapped to business applications. It takes months to clean up the server inventory for every Linux/Unix upgrade initiative.”

According to a literature review of industry best practices, best-in-class asset and configuration management systems are supported by automation tools and software. These automation tools and software supports improved application vulnerability management through the accurate and timely tracking of asset and configuration information. At the time of the evaluation, only the inventory for Red Hat Linux had been automated. The inventory for all other Linux systems and all Unix systems was still based on manual data entry.

3.5 Deficiencies in intake processes affected the efficiency of Linux/Unix service delivery

SSC utilized a number of different processes to manage and execute requests for new services or changes to existing services. These processes included business requests, service requests, and change requests.

• Business requests: Partners and clients could submit a business requirement document (BRD) through SSC’s Enterprise Business Intake (EBI) Process, which then became a business request (BR). The EBI process included various steps from intake to implementation.
• Service requests: A service request (SR) was a request directly assigned to the responsible service line, in which case, it would not follow the EBI process.
• Change requests: A change request (CR) could be initiated to modify an IT service. A change management process was used to control and manage the entire lifecycle of all CRs.

During the evaluation period, the majority of OSMI’s modernization activities utilized the change management process. By contrast, the Linux/Unix division’s operations sometimes involved the EBI process. While conceptually these were distinctive processes, there were occasional overlaps. For example, during the process to implement a CR, a BRD could be submitted through the EBI process to enable cost recovery in cases where a new server needed to be purchased. During the interviews, partners and SSC employees expressed concerns about performance that sometimes conflated BRs and CRs.

According to SSC employees, intake through these different processes was cumbersome and inefficient. The documents used in these processes could contain inaccurate and incomplete information. For example, as the first step of the EBI process, a BRD had to be submitted by partners. Given that SSC’s service catalogue contained outdated and insufficient information, it was often difficult for a partner to specify its requirements.

“If our services were clearly defined and I had a standard offering, I could cut about 80% of those partner requests because they would be going through a defined process and the client would know what to expect as an outcome. Because my services are not well defined, then we empower the partners to request pretty much everything and anything. Then when we say no, they get frustrated.”

The EBI process required the Client Executive team to work with partners to ensure the clarity and completeness of the BRD. However, despite this validation effort, some BRDs still contained inaccurate and incomplete information. SSC interviewees reported that often, the Linux/Unix service lines could not respond to a BR due to improper information in the BRD. This led to extensive follow-ups and clarifications. In some cases, the solution to meet the BR had to be redesigned and costed again, resulting in further delays.

SSC’s IT Service Management (ITSM) tools such as Enterprise Control Desk (ECD) and Onyx were used to manage SRs and CRs. In 2022-23, the Linux/Unix Division shifted from ECD to Onyx for most partners. As an ITSM tool, Onyx was intended to ensure workflow in change management by providing a structured platform to manage the entire change process. It would help simplify workflows by allowing teams to follow defined steps and minimize disruptions while implementing changes.

During the evaluation period, SSC employees reported deficiencies in the use of Onyx as well as in the change management process. For example, as one of the initial steps, a Work Intake Form (WIF) had to be created collaboratively by SSC and the partner. Interviewees reported that the WIF sometimes contained insufficient information. This was caused by the fact that some partners lacked the required technical or corporate knowledge to define the required services. SSC also faced difficulties engaging with the multiple service lines required to complete the form.

The CR assessment stage offered another opportunity to ensure that the required service lines were engaged. For some CRs where up to 8 service lines could be involved, the need to obtain approval from each service line caused delays. The business standard for completing a CR assessment was 5 days. Interviewees reported that this standard was not always met.

These deficiencies could be further compounded by SSC’s lack of internal coordination. In interviews, multiple partner departments reported that their CR tickets created in Onyx had been assigned to the wrong SSC team. This required extensive back and forth between different service lines to reassign and redirect the ticket, slowing down the overall response time. In some instances, after the request finally reached the Linux/Unix service lines for implementation, there was a lack of information on the history of the request. This meant that the Linux/Unix service lines could not easily reference past interactions, changes executed, or strategies considered, which they considered essential for understanding the context of the request. SSC employees reported that this could lead to duplicated efforts and disjointed actions. This further hindered SSC’s capability to properly manage workload and allocate resources.

An effective intake process would ensure the collection of essential information from partners, including new and inexperienced partners. SSC’s partners consistently raised concerns about the delays caused by the various intake processes. In the survey of partners, only 26% of respondents believed that the process from BRD submission to implementation for Linux/Unix hosting services was efficient. SSC employees involved in the EBI process attributed delays to either insufficient information from partners, or the inability of service lines to coordinate their work effectively.

Regardless, partners reported that there was a lack of visibility into the status of their requests. This hindered their ability to properly plan and allocate resources. Similarly, only 29% of respondents believed that SSC’s ticketing systems (including Onyx, ITSM, etc.) for the delivery of Linux/Unix hosting services were efficient.

3.6 Existing financial tools and related processes did not effectively support accurate pricing, cost recovery and transparency

As the GC’s common IT service provider, SSC received funding from partners for many of its services through cost recovery agreements. During the evaluation period, the cost recovery model was intended to support the sustainability of SSC's operations and ensure that funding was available to maintain and improve services to partners. In practice, branch operations were funded at a higher level through appropriations which were allocated to branches through the budget allocation exercise. This exercise provided initial funding at the beginning of the year and subsequent funding throughout the rest of the year. The allocations in this exercise were based on cost-recovery forecasts developed in collaboration with the service lines and CFOPB.

During the evaluation period, the Enterprise Price Estimation Tool (ePET) was the main tool that SSC used to establish the price for individual cost recovery agreements with partners for Linux/Unix services and specific business requests (BRs). This macro-enabled Excel-based application was used to create cost estimates using the main direct costs (e.g., servers, licenses), some indirect costs (e.g., salaries), and ongoing operations and maintenance costs (e.g., support, patch management).

Estimating the costs for Linux/Unix services was challenging. Despite diligent efforts to update and track relevant direct costs, much of the other cost information was incomplete. Many of the processes and steps in the provision of Linux/Unix services were also complex. This made it difficult to create accurate cost estimates, even if all direct and indirect input costs were included.

Client executives and other SSC users developed customized price estimates to address unique client needs. SSC staff reported that this option within the tool was cumbersome and prone to user errors.

The reliance on spreadsheets as a primary tool for cost estimation for projects led to a set of challenges that impacted accuracy and reliability. Specifically, the ePET lacked advanced features to accurately capture the nuances of complex IT programs and generate precise price estimates. For example, hosting solutions often involved multiple service lines such as Linux/Unix, Windows, and Virtualization. However, users reported that the ePET did not easily support the distribution of costs across multiple cost centres in a single estimate.

Additionally, certain types of hosting services defaulted in the tool to a particular cost centre. While the practice of using a default cost centre could streamline the financial processes in some scenarios, it also increased the risk of human error. As this default field needed to be manually adjusted for each estimate, the step could be overlooked affecting the quality of data collected.

Infrequent updates also hindered the effectiveness of the tool. Specifically, the cost information in the ePET was only fully updated every two years. In the interim, users would have to make ad hoc requests for any off-cycle updates. Overall, the price update practice had drawbacks in a fast-moving technological environment where costs could fluctuate significantly. Infrequent updates failed to capture the real-time variations in costs caused by external factors such as inflation, supply chain issues, or shifts in demand. As a result, cost estimates became outdated after each update. If a significant price change occurred shortly after the update, it might only be reflected in two years time at the next scheduled update. Underestimated costs meant that SSC could undercharge partners, leading to budget shortfalls where costs incurred would not be fully recovered.

The lack of a cost validation step in the post-service delivery phase was another gap. Cost estimation during the early stages was crucial for managing resources, but validating these estimates after service delivery was equally important for ensuring accuracy. Without cost validation after service delivery, there was no confirmation that partners had been charged the appropriate amount for the services that they had received.

Pricing and cost recovery was an integrated business process that was based on a decentralized model with multiple branches involved. Coupled with the use of the tool, this led to significant discrepancies. The Linux/Unix Division took the initiative to establish a small team to review and correct certain costing issues. They proceeded to action corrections in ePET and the Business Intake Tracking System (BITS).

At the time of the evaluation, the Linux/Unix Division had identified approximately $12 million in data entry errors and missing costs since 2020 for their two major partners.

These two partners represented 29% of the division’s business portfolio. While the team worked with their Financial Management Advisor to rectify issues, these overall numbers have not been independently validated by CFOPB.

Examples of corrections included:

• Missing servers not quoted in the ePET, thus never charged to the partner;
• Ongoing costs, such as support, missed in the ePET;
• The use of incorrect cost centres between Linux/Unix and other service areas;
• Conflicting costs entered in the various systems (e.g., cost from the ePET appeared as a different amount than in the BITS).

Linux/Unix Division staff believed their efforts to correct cost centre errors affected the funding available to the division. While the ePET may have required cost centre information to be entered, this data was not used. Instead, budget allocations were completed at a higher level and not directly linked to actual BR amounts flowing to individual cost centres.

During the evaluation period, CFOPB was accountable for the integrity of the ePET tool. It provided written instructions on its use and training videos for users. More recently, it has been requiring all SSC managers with delegated authority to take mandatory training because the cost centre owners are accountable and responsible for ensuring their teams perform correct data entry for expenditures.

It is important to note that SSC’s financial accounting system (Sigma) did not link to individual BRs. This made it difficult to validate costs after delivery.

Implications for the modernization of hosting services

Partners indicated that, due to fiscal restraints, the availability of funding often affected their ability to move forward with SSC modernization initiatives. Many partners were coping with technical debt, rising costs, and lowered funding across their information systems and application portfolio. As a result, new investments in modernizing Linux/Unix operating systems put an extra strain on already tight budgets.

“Just breaking up how you (SSC) arrive at the cost would help to give you (SSC) some credibility.”

In this context, some partners reported that Linux/Unix prices seemed high. Others recognized that SSC’s high cost reflected the high levels of security required to meet the GC standards.

Regardless, partners repeatedly and independently reported that the cost estimates provided by SSC for Linux/Unix related services were inconsistent and lacked detail. This was confirmed in the survey of partners. Specifically, only 13% of respondents agreed that SSC’s pricing for Linux/Unix hosting services was transparent. Partners also reported that the lack of transparency had contributed to their reluctance to commit effort and resources towards SSC modernization objectives.

Implications for continuous improvement

During the evaluation period, the financial accounting system (Sigma) was not integrated with BITs and it did not include the costs to deliver individual BRs. SSC relied primarily on budget allocations at the branch level (based on appropriations and forecasted annual cost-recovery revenues) to control costs in the department.

Looking forward, cost validation and cost analysis will likely be increasingly important for continuous improvement and cost containment. In a period of fiscal restraint and fixed budgets, there will likely be an increasing need for SSC executives and managers to review their operations for inefficiencies and identify opportunities for incremental cost-savings at the cost centre level.

Unfortunately, it is not possible with the current suite of tools and available cost information for these executives and managers at the cost centre level to:

• Conduct sensitivity analyses (e.g., determine the financial impacts of changing support costs for aging operating systems);
• Determine with high levels of granularity where cost efficiencies could be made (e.g., determine how much is being spent on support costs for non-standardized operating systems);
• Support divisional planning (e.g., the financial implications of shifting to a new version of an operating system);
• Understand the financial implications of unexpected events (e.g., an unexpected outage or failed update);
• Support lessons learned as expenses are tracked year-over-year.

3.7 Significant talent acquisition and retention obstacles existed for Linux/Unix expertise

According to Canada's 2021 Digital Government Strategy, the GC has struggled with recruiting and retaining skilled digital talent to deliver needed services. SSC’s 2024–25 Integrated Business Plan also highlighted the importance of workforce capacity and retention. As the department responsible for delivering digital services to the GC, SSC has faced ongoing challenges in securing and retaining necessary people and needed skill sets to achieve its operational goals.

Specifically, for Linux/Unix, these challenges are particularly acute. During the evaluation period, Linux/Unix expertise was considered to be a highly specialized and niche talent, with no formal post-secondary education programs producing a new generation of Linux/Unix experts. Most Linux/Unix professionals at SSC acquired their expertise through a series of relevant work experiences, coupled with vendor training courses and industry certification programs. It often took many years to become proficient.

During the evaluation period, the marketplace for Linux/Unix expertise was very competitive. Public sector entities such as SSC had to compete with private sector employers to attract and retain these personnel. SSC competed directly with industries, such as the banking industry, that were able to offer more competitive compensation packages.

Adding to SSC's recruitment challenges was the requirement for on-site or hybrid work arrangements, which was in contrast to the flexibility of remote work offered by many private sector entities during the evaluation period.

“Highly technical roles are sometimes difficult to staff. Work from the office (hybrid) is sometimes a deterrent as many high tech resources are 100% remote.”

As a result of these recruitment and retention challenges, many positions in the Linux/Unix Division and the OSMI group remained unfilled. As of November 2024, among the 296 established positions on the organizational chart of the Linux/Unix Division, 61 were vacant, representing 21% of the Linux/Unix workforce.

To mitigate the difficulties in attracting and retaining employees with Linux/Unix technical expertise, SSC has relied heavily on external consultants to deliver services.^{Footnote 2} Although effective in the short term, this approach did not provide a sustainable solution for building long-term internal capacity.

In addition, SSC hiring managers reported that the existing staffing processes were too generic to effectively address the specific requirements of Linux/Unix technical roles. The generic job description and statement of merit criteria used for staffing Linux/Unix technical positions demanded a broad range of IT experience, but failed to highlight Linux/Unix as a primary competency. Furthermore, there was also no stipulation for the extensiveness of experience required in Linux/Unix operations. Due to this, the merit criteria would not be able to filter for candidates with the depth of proficiency required to perform complex Linux/Unix operations at SSC.

Finally, SSC hiring managers expressed concerns regarding bilingualism requirements for Linux/Unix positions. They recognized the importance of bilingual capacity for client-facing roles. At the same time, they believed that the department’s application of the policy across all positions may have been overly rigid. Specifically, the actual work for some non-supervisory positions involved very limited interactions with partners. The application of a one-size-fits-all approach may have inadvertently narrowed the pool of potential candidates with appropriate technical qualifications.

As part of the evaluation, an independent bilingualism review was commissioned to examine the IT positions for Linux/Unix services. This included the IT-01, IT-02, IT-03 Technical Advisor, and IT-03 Team Leader positions. The review examined job descriptions, organizational charts, position location information and linguistic requirements, descriptions of supervisory responsibilities, legislation, Treasury Board policy documents, Office of the Commissioner of Official Languages publications, and Public Service Employee Survey results. The review included interviews with hiring managers and SSC Official Languages representatives.

The review confirmed that SSC has done a very good job of meeting the TBS expectation that approximately 30% of positions be bilingual. Among the 254 IT positions for Linux/Unix services, 43% had bilingual profiles.

The review also confirmed the challenges in attracting and retaining Linux/Unix expertise within SSC due to bilingual requirements. These challenges particularly affected the recruitment and retention of IT-03 Technical Advisor positions.

During the evaluation period, the IT-03 Technical Advisor played a critical role in delivering services and modernizing operating systems. As of November 2024, these positions made up 34% of all IT positions for Linux/Unix services. As shown in Table 1, 33% of the 85 IT-03 Technical Advisor positions for Linux/Unix had bilingual profiles. The bilingual positions also had high vacancy rates. Among the BBB positions, 33% were vacant. For the CBC positions, 40% were vacant.^{Footnote 3}

According to the bilingual review, the IT-03 Technical Advisor positions did not have supervisory responsibilities and their work duties involved little to no interaction with clients. This meant that they had negligible “central service” demands.

The review highlighted the need for SSC to adopt more flexible staffing processes to meet the recruitment and retention needs of the Linux/Unix specialists. In particular, SSC may wish to consult with other federal organizations that are able to have flexible language requirements. In certain circumstances, organizations such as the Canadian Space Agency, Health Canada, and Natural Resources Canada have been able to obtain exceptions for their language requirements and the use of generic staffing processes.

3.8 Existing performance measurement framework was inadequate to measure the impact of performance

In accordance with the Treasury Board Policy on Results, a Performance Information Profile (PIP) was established and maintained for the Data Centre IT Operations program. It outlined program outputs, outcomes, and performance indicators for the whole program.

According to the Policy on Results, program outcomes are changes or consequences that are attributable to the direct services or products of the program. By contrast, program outputs are the direct services and products that come from the program activities. Outputs are usually within the control of the organization itself.

During the evaluation period, the Data Centre IT Operations PIP identified the following immediate and intermediate outcomes for the program:

• Immediate outcome: Data Centre Services are available to support partners’ and clients’ critical and non-critical applications and services.
• Intermediate outcome: Enterprise solutions and standards are adopted by partners and clients.

“Data centre services being available” directly stemmed from activities under the control of the program. It did not measure the impact of SSC’s data centre services on partner organizations. By definition, this means that it was an output, rather than an outcome.

To measure these identified outcomes, the Data Centre IT Operations PIP identified the following performance indicators.

• Percentage of mainframe computers migrated from legacy data centres to end-state enterprise data centres
• Average time to restore mainframe services from critical incident
• Percentage of time the mainframe services infrastructure is available
• Percentage of time the integrated high performance computing (IHPC) infrastructure is available
• Percentage of time the storage infrastructure is available
• Percentage of time the file services infrastructure is available
• Customer satisfaction with data centre services

These indicators focused on service availability and migration percentages. Data collected for these indicators did not provide insight into the quality of the migration process, the nature of the incidents, or the performance and efficiency gains from the migration. These performance indicators were not robust enough to sufficiently measure outcomes for the program; at most, they measured outputs.

The Linux/Unix Division utilized its own performance indicators outside of the PIP. These indicators measured the percentage of time that operating systems were working and the progress towards upgrading partners to the latest versions of Linux/Unix. Once again, these indicators were not robust enough to measure outcomes and the impact of excellent or poor service by the department.

At best, the Linux/Unix Division used customer satisfaction as a proxy to measure the outcomes arising from its services. At a conceptual level, however, using customer satisfaction to measure outcomes was problematic. Customer satisfaction ratings could be vague, especially if based on perceptions and affected by expectations. The rating could also reflect satisfaction with outputs, rather than outcomes. As a result, customer satisfaction did not directly capture the real impacts of excellent or poor performance for the Linux/Unix program component.

Instead, the program should have measured the ways in which the Linux/Unix Division supported partner and client CIOs in delivering on their mandates.^{Footnote 4} The PIP did not include any criteria to measure how the hosting services contributed to achieving enterprise and stewardship outcomes for the GC. In all cases, there should be performance baselines and target levels of improvement.

These types of performance measurement challenges were observed across all SSC Performance Information Profiles.

4. Conclusions and recommendations

The evaluation identified factors that both supported and hindered the modernization of hosting services.

Partners appreciated the exceptional service they received from the Linux/Unix staff members. This supporting factor contributed to increased confidence among partners in SSC as a service provider and the department’s reputation for quality.

At the same time, there were deficiencies and inefficiencies within SSC that hindered modernization efforts.

Some of these hindering factors were specific to hosting services. First, partners' motivation was affected by the lack of incentive mechanisms, impacting the achievement of modernization objectives and timelines. Second, SSC did not have clear and well-defined modernization strategic and operational plans to effectively communicate its goals and to align modernization efforts. This made it difficult for partners to plan and participate in the modernization process. Moreover, deficiencies in the asset and configuration management system hampered SSC’s ability to effectively implement modernization. It also directly impacted the security and reliability of the GC’s IT infrastructure and services.

There were also department-wide challenges associated with inefficient tools and processes. The deficiencies in SSC’s intake processes caused delays in modernization and had a negative impact on the Linux/Unix Division’s ability to deliver hosting services. In addition, SSC’s existing financial tools were ineffective in supporting accurate pricing, cost recovery and transparency, leading to decreased partner satisfaction and negative impact on SSC’s reputation as a service provider. For recruitment and retention of Linux/Unix expertise, the use of generic staffing processes and manner of application of language requirements prevented SSC from effectively attracting and retaining qualified professionals with specialized Linux/Unix skills. Lastly, SSC’s performance measurement framework for the Data Centre IT Operations program contained poorly defined program outcomes and performance indicators. It did not adequately capture the impacts of excellent or poor performance of the Linux/Unix program component for partners or track the benefits to the Government of Canada from modernization.

Recommendation #1 (HSB): In collaboration with TBS, establish incentive mechanisms to encourage partners to support modernization objectives and comply with timelines.

Recommendation #2 (HSB): Refine and communicate detailed strategic and operational plans for hosting services. This could include:

• Processes for consultation and collaboration with partners;
• Definition of clear objectives and realistic timelines;
• Information on resource requirements for each modernization phase, as this would aid partners in planning their work;
• Specifying feedback channels to allow for clarifications and adjustments.

Recommendation #3 (HSB): Improve the effectiveness of SSC’s IT Asset Management and Configuration Management System. This could include:

• Updating asset and configuration management databases to ensure reliable and up-to-date asset and configuration information;
• Standardizing asset and configuration management processes to ensure consistency and reliability of asset and configuration data;
• Leveraging automated tools and processes to reduce human error and provide real-time updates.

Recommendation #4 (OCSB/HSB): Review and streamline elements of the intake processes to identify and address inefficiencies. This could include:

• Defining, documenting and communicating the processes required for delivering Linux/Unix hosting services through either a business request, service request, or change request;
• Ensuring all essential requirements are collected at intake for decision-making.
• Updating the service catalogue to improve accuracy, relevance, and accessibility of information for partners;
• Improving integration between service management tools (where possible) to enhance visibility of ongoing requests for partners and SSC service lines;
• Streamlining the ordering process for services for partners and update ONYX.

Recommendation #5 (CFOPB): Improve the quality and integrity of cost information within the existing pricing tool to ensure that SSC receives sufficient funds to cover expenses and costs incurred to deliver BRs. This could include:

• Sampling BRs after completion and verifying costs to validate information within the existing tool and improve pricing decisions.
• Improving ePET training materials to reduce user errors. This could include communicating roles and responsibilities for accurate data entry.

Recommendation #6 (HSB): Enhance the recruitment and retention of Linux/Unix expertise through more flexible recruitment processes. This could include:

• Establishing specialized staffing processes for Linux/Unix technical positions on a more frequent basis;
• Reviewing the language requirements for Linux/Unix technical positions to ensure alignment with the position requirements and consider flexibility where appropriate measures are put in place.

Recommendation #7 (HSB): Update the Data Centre IT Operations Performance Information Profile to include meaningful outcome measures and appropriate performance indicators, including baselines and target levels of improvement. The logic model should include service delivery outcomes for partners and stewardship outcomes for the GC.

Appendix A: Additional information on SSC’s Linux/Unix hosting services

Roles and responsibilities of key stakeholders

Several stakeholders played an important role in the delivery of SSC’s hosting services. This included the Treasury Board of Canada Secretariat (TBS) and partner departments. The TBS was responsible for setting strategic direction and guidance for hosting services and to ensure policy compliance. Partner departments were expected to make their hosting decisions consistent with TBS direction and to collaborate with SSC to optimize hosting across the GC. In particular, partner departments were expected to ensure that their applications were modernized, sustainable, and ready to be hosted ahead of launching a request to SSC.

The 2024 Application Hosting Strategy released by the TBS outlined the following roles and responsibilities between TBS, SSC, and partner departments.

Appendix B: Methodology

1. Objective and scope

The evaluation was conducted by the Office of Audit and Evaluation. It represents a snapshot in time reflected in data collected in 2024 and covers the period from 2020–21 to 2023–24. The purpose of the evaluation was to inform decision making by providing a neutral assessment of the responsiveness, effectiveness and efficiency of the modernization of SSC’s Linux/Unix hosting services. The evaluation focused on Linux/Unix services because these services were identified as being a useful case study for all SSC hosting services.

Based on the initial discussion with program management and scoping interviews, the evaluation originally addressed the following key questions.

1. Responsiveness: How effective has SSC been at advancing the enterprise approach from a Linux/Unix hosting services perspective?

2. Effectiveness: How effective have Linux/Unix hosting services been in achieving the intended outcomes?

3. Efficiency: What are the opportunities to improve efficiency?

2. Data collection methods

In order to address the evaluation questions outlined above, the evaluation team employed a mixed-methods approach. Utilizing data triangulation, the team cross-referenced the following lines of evidence to identify, analyze and validate findings.

Literature review

Two literature reviews were conducted to examine relevant themes of Linux/Unix, including industry trends, best practices, and performance measures for Linux/Unix midrange solutions. The information extracted from the literature reviews were used as a foundation to build a sample for the cross-jurisdictional comparison outlined in the section below.

Document review

The evaluation team synthesized information from various documentation sources related to SSC’s Linux/Unix Services, including:

• SSC Governance Board presentation decks (including Executive Oversight Board, Finance, Investment and Internal Management Board, Operations and Services Board);
• Program documents and presentations;
• Other documents pertaining to the modernization of Linux/Unix hosting services.

This analysis was used to establish knowledge of SSC’s Linux/Unix services and its modernization progress, inform the development of a Linux/Unix logic model to support the evaluation, and design the evaluation matrix. To ensure that recommendations remained current and pertinent, the document review was continuously updated until November 2024.

Administrative data analysis

The evaluation team utilized the following data sources to conduct administrative data analysis across fiscal years 2019–20 to 2023–24:

• GC InfoBase – Government of Canada
• Enterprise Data Repository
• Administrative data obtained from the Hosting Services Branch

This analysis was used to build awareness of issues and trends related to service delivery metrics, understand how program expenditures, revenues and funding have evolved, and develop insight into partners’ perspective on SSC’s modernization of hosting services.

Key informant interviews

The evaluation team conducted semi-structured interviews to gather information on views and experiences, explanations, and factual information about SSC’s modernization of Linux/Unix services from various stakeholder groups. Interviewees were selected to maximize diversity and representativeness. A total of 41 interviews were conducted, including 25 internal interviews with SSC stakeholders and 16 external interviews with partner organizations.

Cross-jurisdictional comparison

The evaluation team commissioned a study to review the experiences, best practices and lessons learned from 11 international and national jurisdictions. The analysis was used to compare different jurisdictions to understand their strategies for the management and delivery of hosting services. A special emphasis was placed on international and national perspectives associated with providing hosting services with an enterprise approach.

Bilingualism review

The evaluation team commissioned a study to review the language requirements of the IT positions within the Linux/Unix Division. The study reviewed relevant human resources documents such as job descriptions, organizational charts, as well as related TB policy documents to assess the alignment between language requirements and job duties for these IT positions. Interviews were also conducted with representatives from the Linux/Unix Division and the Official Language Unit within SSC’s Human Resources Services. The study also helped to identify the current work expectations and the possible human resources requirements that these expectations may create in the present and in the future.

Client survey

The evaluation team administered an online survey to capture partner perspectives of the modernization of Linux/Unix services. The survey collected data from partner organizations between May and June of 2024. Out of the 31 partner organizations invited to participate in the survey, 26 provided a response, representing a response rate of 84%.

Appendix C: List of acronyms

BR

Business Request

BRD

Business Request Document

CFOPB

Chief Financial Officer and Procurement Branch

CSFI

Customer Satisfaction Feedback Initiative

CIO

Chief Information Officer

ePET

Enterprise Price Estimation Tool

GC

Government of Canada

HSB

Hosting Services Branch

IT

Information technology

ITSM

IT Service Management

OSMI

Operating System Modernization Initiative

PIP

Performance Information Profiles

SSC

Shared Services Canada

TBS

Treasury Board of Canada Secretariat