Evaluation of Shared Services Canada’s Cloud Services

Executive summary

The Office of Audit and Evaluation conducted an evaluation of SSC’s Cloud Services. The evaluation assessed alignment, effectiveness and efficiency. This report is based on a snapshot of data collected from April to October 2023; it focuses on the Cloud Brokering Service (launched 2017) and the Cloud Advisory Service (launched 2022).

Overall and considering its mandate at the time of data collection, SSC’s Cloud Services were found to be aligned with the enterprise approach, effective in achieving desired outcomes, and generally efficient. However, there were areas where the services could be improved.

It is important to note that after the data were collected, TBS announced significant changes to the GC Cloud Strategy in November 2023. This new strategy was intended to allow for improved oversight of GC cloud usage and implementation of cost and consumption controls. Going forward, the light-touch brokering model will be transformed into a more centralized delivery model. SSC will oversee hosting services for the entire GC, including both cloud and data centres. Although this report is based on a delivery model that is changing, the findings and lessons learned will be relevant for the development of the new program design.

Key findings

During the period under review, there were a number of environmental and contextual factors that influenced the delivery of SSC’s Cloud Services and the achievement of SSC program outcomes, GC outcomes, and individual objectives for partners and clients. These included roles and responsibilities, the need to interact with SSC’s traditional IT service delivery model, and the need to build cloud awareness and expertise across the department.

Roles and responsibilities among GC stakeholders were generally clear, but there was some ambiguity with regard to the procurement of Software as a Service (SaaS). Within SSC, a clearer understanding of roles and responsibilities could help streamline processes (for example, between the cloud team and the service lines that provide services in traditional areas of IT infrastructure, like security or networks).

Partners and clients appreciated SSC’s Cloud Services’ innovative program design. SSC centralized the procurement of cloud services through framework agreements with major cloud providers, which enabled departments and agencies to build their cloud environments independently. Most users benefited from the rapid procurement through the GC Cloud Services Portal, which took 2 days on average. However, about one-third of survey respondents had to submit a business requirement to enable their cloud environments. This meant that they had to go through SSC’s traditional delivery model. The average completion time for all cloud-related requests was 278 days. The standard SSC business requirement process was complex and lengthy, and resulted in reduced agility for partners and clients.

During the evaluation, it became clear that SSC needed to build cloud awareness and deepen cloud expertise, particularly within teams who were not directly involved in cloud service delivery. SSC could adopt a standardized approach to educating employees across all branches about cloud technologies. Equipping employees with the necessary skills will lead to more effective work in an environment that increasingly merges traditional IT services with the cloud.

In terms of achieving desired program outcomes, SSC’s Cloud Services contributed to fostering an enterprise approach, but alignment and standardization could be enhanced. Specifically, the program made progress in the consolidation and greater standardization of GC IT service offerings. At the same time, the light-touch broker design still allowed departments and agencies to increase their autonomy and pursue non-standardized approaches to cloud adoption. This represented a step away from GC centralization.

Other SSC-desired outcomes were to enable partners to make smart decisions about cloud adoption and to accelerate cloud adoption across the GC. SSC’s Cloud Services provided guidance and expertise through the cloud documentation portal and the Cloud Advisory Service. While only 19% of survey respondents reported using the Cloud Advisory Service, this was not surprising, given its recent launch in 2022. However, consistent misunderstandings among partners and clients around the basic design of cloud services (for example, the procurement thresholds or the optional nature of the services), as well as the lack of awareness of available cloud services and documentation, highlighted the need for more effective communications.

SSC’s Cloud Services were intended to contribute to broader GC cloud outcomes, including evolving the service-focused culture of the GC, maximizing business value and reducing accumulated technical debt. SSC contributed by enabling access to cloud services and facilitating the deployment of critical business applications. This was especially important during the COVID-19 pandemic, where rapid deployment was critical. At the same time, some partners and clients attributed these benefits to cloud providers (or, more broadly, to cloud technology) rather than to SSC’s Cloud Services. The lack of available evidence to demonstrate SSC’s contribution to broader GC Cloud outcomes highlighted the need to document and further demonstrate SSC’s Cloud Services value in achieving broader GC cloud outcomes.

Partners and clients had their specific objectives for adopting cloud technology. These objectives included application modernization and optimization, increased agility and increased efficiency in serving end users. SSC’s Cloud Services supported the achievement of these cloud-related objectives through centralized procurement and improved security. However, partners and clients also raised concerns about the cloud funding model, the brokering fee, and a lack of transparency. While the overall user experience with the Cloud Services Portal was positive, some aspects could be improved. Challenges with the portal included limited functionality and inefficiencies, which generated additional work.

Finally, the foundational performance measurement deliverables were not adequate for measuring outcomes or capturing the impacts of poor or excellent SSC performance. The key performance indicator was narrow in scope and based on a modest target that SSC had consistently achieved over the last three years. The CSFI served as a feedback mechanism, but it was not a reliable indicator for measuring the achievement of business outcomes for partners and clients or evaluating the ways in which SSC supported partners and clients in delivering on their mandates. SSC’s Cloud Services Performance Information Profile lacked a robust logic model to depict the relationship between program objectives, activities, outputs and intended outcomes. Finally, outcomes and performance measures related to the enterprise approach (now Delivering Digital Solutions Together) and the GC Cloud Adoption Strategy were notably absent.

Recommendations

Given the changing nature of the program, this evaluation includes 6 recommendations that are likely to remain relevant in the new program design. It also identifies 9 opportunities for improvement and 9 insights from the experiences of other jurisdictions for consideration.

Recommendation 1: Identify and address critical barriers to delivering cloud-related business requests

• Develop and implement an action plan to identify and address critical barriers to efficient service delivery and implement improvements to processes that align with the new design of SSC’s Cloud Services.
  • Office of Primary Interest: The Assistant Deputy Minister of Hosting Services Branch in collaboration with Operations and Client Services Branch and Citizen Business, National Defence and Policing Branch as necessary

Recommendation 2: Build cloud awareness and expertise across SSC

• Develop and implement an action plan to increase cloud awareness within SSC and provide employees (including those not directly involved with cloud) with skills to prepare for and adapt to cloud technology.
  • Office of Primary Interest: The Chief Technology Officer in collaboration with Enterprise IT Procurement and Corporate Services Branch as necessary

Recommendation 3: Clarify SSC’s roles and responsibilities for cloud services

• Collaborate with stakeholders to define and communicate GC-wide roles and responsibilities to partners and clients (for example, this could include how they have changed or will change under the new program design)
• Develop an internal SSC RACI that identifies branch roles and responsibilities.
  • Office of Primary Interest: The Assistant Deputy Minister of Hosting Services Branch

Recommendation 4: Develop and implement an ongoing process for monitoring and enhancing the GC Cloud Services Portal user experience

• Review and address requests to make the portal more user friendly.
  • Office of Primary Interest: The Assistant Deputy Minister of Hosting Services Branch

Recommendation 5: Develop a communications plan to ensure that partners and clients understand the program design

• SSC should leverage the launch of the new program to develop a communications strategy to make sure that partners and clients understand the new program design, and to facilitate the onboarding of any new client or partner to this service. For example, this could include clearly identifying which services are optional, clarifying the procurement and threshold process for cloud services, or building understanding of advisory services. The plan could also aim to streamline communication delivery methods to ensure relevance and tailor information to users’ needs. This could include taking into account organization and employee characteristics or exploring the use of AI chatbots and online help.
  • Office of Primary Interest: The Assistant Deputy Minister of Hosting Services Branch in collaboration with Strategy and Engagement Branch as necessary

Recommendation 6: Update and enhance program performance measurement

• Revise the Performance Information Profile to track progress toward objectives that align with the new program design. This would include revised performance indicators, baselines, realistic target levels of improvement, and a new logic model that distinguishes among objectives, activities, outputs and outcomes (immediate, intermediate and ultimate). It should include outcomes related to both cloud service delivery and the delivery of enterprise-level services.
  • Office of Primary Interest: The Assistant Deputy Minister of Hosting Services Branch in collaboration with Strategy and Engagement Branch as necessary

1. Introduction

This report presents the results of an evaluation of Shared Services Canada’s (SSC’s) Cloud Services in the Hosting Services Branch. In accordance with the Policy on Results, this evaluation provided a neutral assessment of the alignment, effectiveness and efficiency of SSC’s Cloud Services to inform decision making. The evaluation was managed internally by the Office of Audit and Evaluation. It represents a snapshot in time of data collection from 2023, covering the period from fiscal years 2018-19 to 2023-24.

It should be noted that significant changes to the program are set to come into effect in fiscal year 2024-2025. The new GC strategy, expected to be released in 2024, will focus on choosing a hosting model for applications that best benefits the long-term needs of each workload. It is intended to allow for improved oversight of GC cloud usage and implementation of cost and consumption controls. All application hosting operations, including cloud and data centres, will be centralized at SSC.

For easy reading, we invite the reader to consult the list of acronyms at the beginning of this report, as well as the list of definitions in Appendix A (in the report, defined words or expressions include a hyperlink to this appendix). In addition, Appendix B offers information on the methodological approach used to carry out this evaluation.

The report first presents a description of the program profile (section 2). Additional details on the program are available in appendices C and D. The report then moves on to the key findings of the evaluation, starting with those concerning SSC’s environment and the context influencing the delivery of cloud services (section 3).

Findings related to outcomes are organized around program objectives (see the notional logic model developed to support this evaluation in Appendix E). The outcomes resulting from the objectives that SSC had set for its cloud services are presented in section 4. Section 5 provides the results related to SSC’s contribution to the GC Cloud Strategy. Lastly, section 6 presents findings related to SSC’s contribution to the achievement of partner and client objectives and their alignment with SSC’s objectives.

Section 7 reviews the performance measurement framework that SSC had developed to monitor progress toward achieving desired outcomes. The final section (section 8) presents conclusions and recommendations.

Where relevant, each section of this report includes opportunities for improvement that may be useful in developing the new cloud service model. These opportunities are not recommendations. Rather, they are ideas that have emerged from data collection; they are reported for consideration only. Recommendations focus on the parts of the services that will likely remain under the new program design. Recommendations will require the Hosting Services Branch to develop management responses and action plans to address these issues. The implementation of these action plans will be monitored on an annual basis and their status reported to senior management.

2. Program profile

This section provides background information on the Cloud Brokering Service and the Cloud Advisory Service prior to presenting the evaluation results.

2.1. Background

SSC’s Cloud Services were launched in 2017 following the GC’s release of the Right Cloud Adoption Strategy the previous year. In 2018, this GC strategy was updated to the Cloud First Adoption Strategy, and then evolved to “cloud smart” in 2023. This encouraged smart cloud adoption across the GC and ensured that applications and workloads were hosted by the right solutions. The GC Cloud Adoption Strategy identified 3 ultimate outcomes for cloud adoption:

• Evolve the service-focused culture of the GC by delivering client-centric cloud services
• Reduce the GC’s technical debt by reducing the amount of aging technology
• Maximize business value by enhancing the GC’s service efficiency and quality

As a broker delivering SSC’s Cloud Services, SSC committed to being a leader in the cloud space for the GC by enabling smart cloud adoption and facilitating the realization of cloud benefits. To achieve this, SSC identified the following objectives:

• Foster SSC’s enterprise approach by strengthening enterprise management, visibility, security monitoring, and integrated service management
• Accelerate cloud adoption by enabling the use of standardized, secure and smart approaches
• Enable access to secure and reliable cloud services for any cloud deployment model by establishing a self-serve “one-stop shop” Cloud Services Portal

To meet these objectives, SSC’s Cloud Services have aimed to achieve the following program outcomes:

• SSC’s Cloud Services foster an enterprise approach by delivering cloud services that are consolidated, standardized and secure
• SSC’s Cloud Services facilitate smart cloud adoption by providing access to knowledge and expertise
• SSC’s Cloud Services accelerate cloud adoption by providing access to services that enable agility

During the evaluation period, SSC’s Cloud Services were authorized to provide a Cloud Brokering Service (launched 2017) and a Cloud Advisory Service (launched 2022). The services were available to federal departments and agencies as well as other levels of government.

2.2. The Cloud Brokering Service

Through the Cloud Brokering Service, SSC acted as a light-touch broker between departments and cloud providers. SSC developed framework agreements with 8 public cloud providers^{Footnote 1} and created a Cloud Services Portal to provide departments and agencies with access to purchase secure cloud services directly. The portal included 3 procurement vehicles:

• The Government of Canada Cloud Framework Agreements, in which partners and clients could purchase secure cloud services from approved cloud providers
• The GC Cyber Security Procurement Vehicle, in which partners and clients could purchase solutions, technologies and capabilities related to cyber security
• The Government of Canada Cloud SaaS Method of Supply, in which partners and clients could procure client-focused SaaS applications and associated professional services

In October 2021, the portal expanded to include a cloud documentation portal, which provided departments and agencies access to in-depth information about cloud adoption in the GC.

Cloud providers offered standard cloud services (such as identification and security, analytics and big data, database management, networking services and platform services) and, to a lesser extent, training, advisory and support services. To procure cloud services through SSC, departments and agencies had 3 options, depending on the procurement dollar amount (Figure 1). Importantly, if a department or agency generated a contract within a specific threshold, but reached a point where its cloud consumption was in danger of exceeding that threshold, it had to recompete the additional cloud business requirements under a higher-valued threshold category.^{Footnote 2}

Figure 1: Procurement thresholds in the Cloud Framework Agreements

Figure 1 - long description

Figure 1 is a graphic that illustrates the 3 threshold categories for cloud procurement under the procurement framework agreements. For procurements less than $500,000, customers could enter a direct contract with their preferred cloud provider. For procurements between $500,000 and $4.5 million, customers had to compare a minimum of 3 cloud providers and choose the provider with the best contracting price or the best value for their requirement. For procurements more than $4.5 million, and up to $37 million, customers needed to complete a requirement confirmation request with all 8 cloud providers within the framework agreement.

Alternatively, departments and agencies could procure cloud services independently if their requirements fell within their individual contracting authority. For most departments, authority was under $200,000 for non-competitive contracts and under $3.75 million for competitive contracts. If their requirements fell outside their authority, they would have to procure through SSC or PSPC, depending on the expected contract value. By opting for independent procurement, organizations lost the ability to use SSC’s framework agreements.

2.3. The Cloud Advisory Service

SSC’s Cloud Advisory Service was launched in 2022 to help guide departments and agencies through their cloud adoption journey by sharing knowledge and expertise to enable departments and agencies to improve their understanding of cloud technologies. This advice included information on cloud security and design, and the enablement and implementation of partner and client cloud environments.

2.4. Program roles and responsibilities

There were several organizations involved in the delivery of SSC’s Cloud Services, including partners and clients, the Secretariat of the Treasury Board of Canada (TBS), the Canadian Centre for Cyber Security (CCCS), Public Services and Procurement Canada (PSPC), and cloud providers (Table 1). See Appendix D for more detailed roles and responsibilities.

Table 1: Main responsibilities of each organization for SSC’s Cloud Services

Stakeholder	Main responsibilities
SSC	Provided light-touch cloud brokering services Acquired commercial cloud services Created and managed customer cloud accounts, including intake of orders and requests via the Cloud Services Portal and invoicing customers Provided cloud advisory services
Departments and agencies consuming cloud services	Developed departmental governance for cloud services and cloud adoption frameworks Decided on deployment and service model selection Continuously monitored and managed cloud services
TBS	Developed GC governance, strategy and corporate policy for cloud services, including oversight and risk assessment of cloud service requests
CCCS	Created the Cloud Security Assessment ProgramFootnote 3, which conducted assessments to ensure that cloud service usage in the GC was secure Monitored activity of customer deployments in contracted cloud services
PSPC	Established supply arrangements with prequalified cloud providers for Software as a Service (SaaS) offerings, and assessed physical security controls of cloud providers
Cloud providers	Provided pre-approved cloud services to GC customers and operated and supported the services

2.5. SSC’s Cloud Services spending

SSC’s Cloud Services received allocated funding from various sources to cover program expenditures (including A-base and allocated revenues^{Footnote 4} ). Total funding and expenditures for SSC’s Cloud Services have increased over 5 years (Table 2).

Table 2: Cloud Services allocated funding and expenditures by fiscal year 2018-2019 to 2022-2023

	2018-2019	2019-2020	2020-2021	2021-2022	2022-2023
Allocated funding
	$9,503,000	$15,140,000	$24,218,000	$25,740,000	$29,292,000
Expenditures
Salary costs	$4,475,000	$6,782,000	$8,365,000	$7,786,000	$11,398,000
Operational costs	$3,902,000	$6,620,000	$13,607,000	$13,795,000	$17,168,000
Corporate expenditures	$1,725	$30,000	$107,000	$44,000	$41,000
Total expenditures	$8,379,000	$13,432,000	$22,080,000	$21,624,000	$28,607,000
Slight variances in the totals are due to rounding

To support some operational and salary costs, TBS approved brokering fees as a revenue source. These fees primarily supported the cloud brokering, strategy, stewardship, and security teams under the Cloud Product and Management Services Directorate (CPMSD). The funds were also used to support portions of the Enterprise IT Procurement team, who worked closely with the brokering team on cloud procurement.^{Footnote 5} SSC charged a brokering fee of 10% of the costs incurred by GC departments and agencies for cloud services purchased through the framework agreements. Cloud consumption and related collected brokering fees increased significantly over time (Table 3).

Table 3: Cloud consumption and associated brokering fees by fiscal year 2019-2020 to 2022-2023^{Footnote 6}

Fiscal year	Total cloud consumption	Consumption subject to brokering feeFootnote 7	Collected brokering fee
2019-2020	$1,396,000	$1,395,000	$140,000
2020-2021	$47,721,000	$21,758,000	$2,176,000*
2021-2022	$104,367,000	$97,272,000	$9,727,000
2022-2023	$156,942,000	$146,561,000	$14,656,000
Slight variances in the totals are due to rounding *In the last quarter of 2020-21, brokering fees were suspended

In accordance with the Directive on Charging and Special Financial Authorities, SSC’s Cloud Services were not intended to generate surpluses. The program had several mechanisms to ensure that surpluses were not generated, including suspending the brokering fee during the year if it was expected to exceed the operational costs of the services.

2.6. SSC’s Cloud Services usage

SSC’s Cloud Services were leveraged by 91 organizations across Canada since their launch, including 43 federal partners^{Footnote 8}, 40 mandatory and optional clients and 8 organizations within other levels of government. Organizations invested in cloud technology at varied rates. For example, cloud consumption in fiscal year 2022-2023 ranged from less than $1,000 to more than $19 million. The highest-consuming customer of SSC’s Cloud Services was Communications Security Establishment (CSE), followed by Statistics Canada. Each paid an average annual brokering fee to SSC of more than $700,000.

Since fiscal year 2018-2019, 38% of SSC’s cloud customers were considered “very high spenders,” spending more than $1 million on cloud in total (Figure 2). A similar 37% were considered “high spenders,” spending between $100 thousand and $1 million on cloud in total. Partners were more likely to be “very high spenders” on cloud compared to clients or other levels of government.

Figure 2: Partners spent more on cloud than clients between fiscal year 2018-2019 and fiscal year 2022-2023

Figure 2 - long description

Figure 2 is a modified line graph that shows how many partners, clients, and other customers fall into each spending category for GC Cloud Services. Between fiscal years 2018-19 and 2022-23, partner total expenditures on cloud were more than client total expenditures on cloud.

Type of customer	Very high spenders (more than $1 million)	High spenders (between $100,000 and $1 million)	Moderate spenders (between $10,000 and $100,000)	Low spenders (less than $10,000)
Partner	27	14	2	0
Client	5	16	13	6
Other	3	4	0	1
Source: Cloud fulfillment dashboard, retrieved April 2023

3. Environment and context influencing SSC’s Cloud Services

3.1. Roles and responsibilities needed greater clarity

3.1.1. Roles and responsibilities among stakeholders were mostly clear

SSC relied on the support of several stakeholders to deliver services (see Appendix D). Overall, SSC employees, partners and clients reported that roles and responsibilities between the different organizations were clear. Procedures and processes were also clear and well established, and relationships among TBS, CCCS^{Footnote 10}, PSPC and SSC were collaborative. There was also a RACI matrix that described the roles of each organization for cloud services.

3.1.2. Roles and responsibilities within SSC were not completely clear

3.2. SSC’s structure and culture impacted the delivery of cloud services

3.2.1. SSC’s structure was siloed and operated with conflicting priorities and dependencies

3.2.2. SSC’s traditional IT service delivery was not agile enough to meet expectations for cloud services

3.2.3. SSC’s culture did not sufficiently support internal cloud adoption

In the context of SSC, interviewees expressed concerns that SSC’s overall culture was not ready for cloud adoption. Indeed, SSC – and the GC in general – was late to adopt cloud compared to other jurisdictions, some of whom launched cloud strategies as early as 2011 or 2013 (including the United States, Australia and the United Kingdom). Since their launch in 2017, SSC’s Cloud Services have implemented new processes to embrace a cloud culture and move away from SSC’s traditional service model in order to deliver cloud services effectively. But SSC interviewees reported that SSC was having difficulty updating its traditional processes to embrace this approach.

Some interviewees expressed how difficult it was to change work processes in SSC and that some processes were not ready for or supportive of cloud. In particular, interviewees raised concerns about the Service Authorization process, highlighting its incompatibility with the constantly evolving service catalogue and the potential introduction of new capabilities in the cloud. The Service Authorization process added to the efforts required to reach and mobilize implicated service lines, reach agreement and implement change. This hindered the capacity of Cloud Services to be responsive to client needs.

In an effort to challenge the status quo, SSC’s Cloud Services developed the “simple change” pilot to identify simple requests that did not need to go through the business requirement process. The initiative took over a year to achieve and had yet to be formally authorized department wide, even though it significantly streamlined service delivery and made pathfinders satisfied.

There was an opportunity to support SSC’s transition to a cloud culture through focused cloud upskilling and general cloud education initiatives. SSC interviewees did not believe that the department had a formal approach to build employee awareness and knowledge of cloud technologies across the department, although resources had been made available. A few interviewees reported that there was a widespread lack of understanding of cloud within SSC. Others mentioned that there were even fears about cloud. Teams outside of the cloud directorate reported that they had learned on the job rather than participating in formalized training initiatives. Some searched for knowledge when required through departmental and client collaborations, particularly with the cloud directorate. Other teams had increased their cloud capacity by upskilling their current workforce, hiring new employees or leveraging consultants. Notably, SSC’s Digital Enterprise Skilling Program contributed to the upskilling of SSC employees, with over 270 IT certifications to date. However, there was an opportunity to formalize upskilling initiatives to develop employee cloud-related knowledge and skills more broadly across the department.

4. The achievement of program outcomes

Through various activities, SSC’s Cloud Services were expected to achieve several key program outcomes (see notional logic model in Appendix E). The following section explores the program’s progress in achieving the outcomes below:

• aligning with and fostering an enterprise approach (section 4.1)
• enabling partners to make smart decisions about cloud adoption (section 4.2)
• accelerating cloud adoption across the GC (section 4.3)

4.1. SSC’s Cloud Services progressed toward fostering an enterprise approach, but alignment and standardization could be improved

4.1.1. SSC’s enterprise approach was well understood, but it was unclear how well SSC’s Cloud Services aligned with this approach

4.1.2. SSC’s Cloud Services made progress toward centralizing and standardizing cloud services, but business intake could be further integrated

Figure 5: Survey respondents reported that SSC helped their organization access bulk procurement and enterprise-level contracts, and consolidate and standardize the GC IT service offerings

Figure 5 - long description

Figure 5 is a stacked bar graph that shows the results of 3 survey questions. Most survey respondents agreed that SSC helped their organization access bulk procurement and enterprise-level contracts and consolidate and standardize the GC IT service offerings. There were 99 respondents who answered these questions. Slight variances in the totals are due to rounding.

Survey question	Agree	Neutral	Disagree	Don’t know
SSC’s Cloud Portal helps my organization to access bulk procurement and enterprise level contracts.	59%	17%	8%	16%
SSC’s Cloud Services contribute to the consolidation of the Government of Canada’s IT service offerings.	57%	18%	7%	18%
SSC’s Cloud Services contribute to the standardization of the Government of Canada’s IT service offerings.	53%	15%	8%	24%
n = 99 Source: The Cloud evaluation survey, administered in fiscal year 2023-24, Q2

4.1.3. SSC’s Cloud Services allowed partners and clients more autonomy, but there was room to increase standardization and reduce GC security risks

4.2. SSC’s Cloud Services enabled smart cloud adoption, but attribution to its Cloud Advisory Service was low

Given that advisory services were only launched in 2022, more time was likely needed for partners and clients to see the results materialize from advice provided by SSC.

4.3. SSC’s Cloud Services accelerated cloud adoption, but many aspects of the program were unclear to partners and clients

However, basic aspects of the program were not well understood, and this caused confusion. SSC’s Cloud Services were designed as an optional service. This meant departments and agencies could procure cloud services through the Cloud Brokering Service’s framework agreements or through independent procurement (see section 2.2). Nevertheless, SSC had an exclusive mandate to deliver some cloud services (such as Secure Cloud to Ground, Secure Cloud Enablement Defence, and Firewall). This meant that partners had to procure those services through SSC’s business requirement process. While optional clients had more freedom, the design of SSC’s Cloud Services made it difficult for departments and agencies to see the program as optional. Notably, almost all interviewed partners and clients believed the program was mandatory.

5. Extent to which SSC is contributing to broader GC cloud outcomes

The 2023 GC Cloud Adoption Strategy aimed to achieve 3 ultimate outcomes for cloud adoption (evolving the service-focused culture of the GC, maximizing business value and reducing accumulated technical debt). SSC’s Cloud Services had no direct control over these ultimate outcomes, but aimed to influence and make progress towards their achievement. The following section discusses SSC’s contribution to the achievement of these ultimate outcomes.

However, departments and agencies that were already advanced in their cloud journey benefitted less from SSC services. Notably, a few interviewees attributed responsibility for more efficient service delivery to the cloud providers (or, more broadly, to cloud technology) rather than to SSC. In fact, a few partners explicitly stated that SSC did not help their organization offer better services. Instead, it was cloud that improved services by default of its technology. In addition, fewer than half of survey respondents reported that SSC had helped their organization (Figure 8):

• to maximize business value by increasing efficiency of service delivery (40%)
• to maximize business value by increasing service quality (39%)
• to reduce technical debt (34%)
• to reduce reliance on legacy systems (28%)

Figure 8: Fewer than half of survey respondents reported that SSC's Cloud Services contributed to improving services by departments, maximizing business value and reducing technical debt.

Figure 8 - long description

Figure 8 is a stacked bar graph that shows responses to 4 survey questions. Less than half of survey respondents agreed that SSC’s Cloud Services helped their organization deliver services more efficiently, improve the quality of their services, maximize business value, or reduce technical debt. There were 99 respondents to these questions.

Survey question	Agree	Neutral	Disagree	Don’t know
SSC’s Cloud Services help my organization to deliver services in a more efficient way.	40%	28%	12%	20%
SSC’s Cloud Services help my organization to improve the quality of our services.	39%	21%	14%	26%
SSC’s Cloud Services help my organization to reduce its technical debt.	34%	38%	11%	18%
SSC’s Cloud Services help my organization to reduce reliance on legacy systems.	28%	31%	11%	31%
Source: The Cloud evaluation survey, administered in fiscal year 2023-24, Q2

Consequently, while SSC’s Cloud Services were seen as the conduit connecting organizations to cloud providers, SSC’s value was perceived as secondary to the services themselves. Notably, working-level respondents were more than twice as likely to agree that SSC’s Cloud Services helped improve the quality of their organization’s services (56%) compared to senior executives (25%). This suggested that SSC struggled to tell its success stories and to demonstrate its value to departments and agencies, particularly at more senior decision-making levels.

6. Findings related to individual partner and client objectives

In addition to the SSC and GC cloud objectives, partners and clients identified additional objectives related to their use of cloud technology. This section explores those individual partner and client objectives (section 6.1) and their alignment with SSC’s Cloud Services (section 6.2). It also addresses some issues that emerged from data collection, including billing transparency and the funding model (section 6.2.1), and user experiences with the portal (section 6.3).

6.1. Partner and client objectives and challenges in cloud

6.2. Aligning partner and client objectives with SSC’s Cloud Services

6.2.1. Partners and clients disagreed with the funding model

6.3. Cloud portal user experience was mostly positive, but there were opportunities for improvement

Most interviewees acknowledged that the Cloud Services Portal provided an easy and seamless procurement experience for users. However, a few interviewees commented on the portal’s limited authorization and its inability to support the entire cloud workflow. Similarly, only 43% of users surveyed agreed that the portal made it easy to procure the cloud services their organization needed.

When asked about their user experience in procuring cloud services, departments and agencies reported several challenges. The most common was associated with the forms, which they viewed as cumbersome, complicated, and at times redundant. Also, users could not submit multiple requests at once, cancel or modify an order once sent, view status updates of requests, or view detailed service descriptions from the portal. They reported that this lack of functionality created unnecessary work for users. A few partners also cited a lack of clarity about which portal and marketplace items were pre-assessed up to a security level of Protected B.

7. Assessment of SSC’s Performance Measurement Framework

In accordance with the Treasury Board Policy on Results, the performance information regarding SSC’s Cloud Services is found in the corresponding Performance Information Profile. This section describes the indicators included in this document (section 7.1) and the policy requirements related to performance measurement (section 7.2).

7.1. Current performance measurement metrics for SSC’s Cloud Services were inadequate

The Performance Information Profile identified 2 performance indicators to measure the success of SSC’s Cloud Services in reaching its objectives.

1. Percent of cloud brokering requests fulfilled within established service level standards

Since fiscal year 2020-2021, the cloud brokering team has exceeded their target to fulfill at least 90% of requests within 7 days. Their performance was significantly better than the 7-day threshold established by service level standards (averaging 2 days). This performance indicator was narrow in scope and it excluded cloud business requirements that needed the collaboration of other SSC service lines for completion. This suggests that there is an opportunity to review the current service level standards to better reflect the program’s intent to continuously improve service delivery.

Table 4: Performance measurement indicator - service level standards

Indicator	Target	Frequency	Data source
Total number of requests (all levels of complexity) filled within service level standards divided by the total number of requests (all levels of complexity) times 100	At least 90%	Monthly	Vendor system

2. Customer satisfaction with cloud brokering and advisory services

During the evaluation period, the CSFI also collected feedback on service satisfaction from the Chief Information Officers of 45 partner organizations. SSC’s Cloud Services had a target score of between 3.6 and 5 (Table 5).

Table 5: Performance level indicator - CSFI satisfaction score

Indicator	Target	Frequency	Data source
Average client satisfaction score on a 5-point scale	Between 3.6 and 5.0	Bi-annually	CSFI questionnaire

SSC’s Cloud Brokering Service obtained an overall average of 3.61 out of 5 in the CSFI since October 2018 (Figure 10). Of particular note, after a consistent increase in satisfaction scores, there was a drop in satisfaction after October 2022. Qualitative data analysis revealed a mixed perception of service quality, with partner Chief Information Officers reporting issues related to SSC’s communication, brokering services design and service delivery.

Figure 10: Cloud Brokering Service consistently increased satisfaction scores until 2023.

Figure 10 - long description

Figure 10 is a lollipop graph that shows how SSC’s cloud brokering services performed on the Customer Satisfaction Feedback Initiative (CSFI) service questionnaire from October 2018 to October 2023. It also compares the results against the program’s target score of 3.6. The graph shows that the Cloud Brokering Services consistently increased satisfaction scores until 2023.

CSFI Date	Cloud Brokering Services Score
October 2018	3.60
April 2019	3.35
October 2019	3.38
October 2020	3.49
April 2021	3.65
October 2021	3.71
April 2022	3.76
October 2022	3.95
April 2023	3.67
October 2023	3.55
Source: SSC’s Enterprise Data Repository. Retrieved January 2024

Although the CSFI was limited to brokering services, the feedback also mentioned other cloud-related services submitted through the business requirement process (such as Secure Cloud to Ground or Secure Cloud Enablement Defence). This suggests that some respondents may have lacked a clear understanding of the scope of the brokering services and how it differed from the business requirement process. This could be clarified in the questionnaire to ensure that accountability can be attributed to the brokering service alone.

Satisfaction scores for the Cloud Advisory Service have been collected since October 2022 (Figure 11). In all cases, the average satisfaction rating was well above the target. A more ambitious target would reflect SSC’s intention to continuously improve service delivery.

Figure 11: SSC’s Cloud Advisory Service has consistently met its target score.

Figure 11 - long description

Figure 11 is a lollipop graph that shows how SSC’s cloud advisory services performed on the CSFI from October 2022 to October 2023. It also compares the results against the program’s target score of 3.6. The graphs shows that SSC’s cloud advisory services consistently exceeded its target score.

CSFI Date	Cloud Brokering Services Score
October 2022	4.07
April 2023	3.76
October 2023	3.82
Source: SSC’s Enterprise Data Repository. Retrieved January 2024

7.2. Policy requirements for performance measurement

Programs are supposed to deliver outcomes. Program outcomes are changes or consequences that are attributable to the direct services or products of the program. They should be distinguished from outputs, which, according to the Treasury Board Policy on Results, are direct services and products that come from the activities of an organization. Outputs are usually within the control of the organization itself.

The indicators in the Performance Information Profile were not robust enough to sufficiently measure outcomes for Cloud Services. Of specific concern, fulfilling cloud brokering requests directly stemmed from the activity under the control of the program. This means that the indicator was measuring outputs, rather than outcomes.

At a conceptual level, using customer satisfaction to measure outcomes arising from SSC performance was problematic. Customer satisfaction ratings were a proxy for measuring the achievement of outcomes, and if based on perceptions and affected by expectations, they could be vague. The ratings could also reflect satisfaction with outputs (which would be within the control of the program). As a result, customer satisfaction did not directly capture the real impacts of excellent or poor SSC performance for cloud services and the ways in which SSC supported partners and clients in delivering on their mandates. In addition, the Performance Information Profile did not include any criteria to measure the cloud services contribution to achieving enterprise and stewardship outcomes.^{Footnote 26}

8. Conclusions and recommendations

Overall and considering its mandate at the time of data collection, SSC’s Cloud Services were found to be aligned with the enterprise approach, effective in achieving desired outcomes, and generally efficient. However, there were areas where the services could be improved.

SSC’s Cloud Services will undergo significant changes in fiscal year 2024-25. The program’s light-touch brokering model will be transformed to make way for a more centralized delivery model. Although this report is based on a program design that is changing, the lessons learned and the findings are both relevant and timely to inform any changes to SSC’s cloud program.

8.1. An innovative service design challenged SSC’s traditional service delivery model

The program was initially designed to accelerate the adoption of cloud computing in the GC by consolidating the acquisition of cloud services delivered centrally by SSC. SSC was assigned a light-touch brokering mandate and entered into framework agreements with major providers of public cloud services, putting in place a portal to enable access. Once the products and services had been procured, departments and agencies could build their cloud environments as they saw fit as long as they complied with common minimum security requirements (such as the GC cloud guardrails issued by TBS). This service delivery model was somewhat unique for SSC, particularly when compared to the model for other available hosting services (such as data centres), where SSC had the full mandate for infrastructure and platform development.^{Footnote 27}

The sharing of responsibilities for cloud computing was a distinct departure from SSC’s traditional service model. It is not surprising, therefore, that it generated mixed reactions, both within SSC and among partners and clients. Some SSC employees perceived this as a decentralization of services, resulting in low visibility into partners’ environments. But others were positive about the proposed innovation, particularly because of the speed of delivery that was enabled. Overall, departments and agencies appreciated the ability to access self-serve services and saw great value in the time savings associated with establishing framework agreements with major cloud providers, especially because these framework agreements generally enabled access to services within 2 days.

However, this innovative service design was challenged when interacting with SSC’s traditional service delivery model. A third of survey respondents who had used SSC Cloud Brokering Services also reported requiring more complex services from SSC, which necessitated the use of SSC’s business requirement process. This traditional service model could take months to complete cloud-related business requirements, hindering the agility of these organizations. Given the anticipated growth of cloud, these challenges are likely to grow. In this regard, there is an opportunity for SSC to be proactive and rethink the way it delivers its services to make them more agile. This could provide an opportunity to improve SSC’s overall performance and simultaneously enhance its reputation with partners.

An emerging finding was the need to deepen cloud expertise across SSC. According to interviewees, SSC lacked a standardized approach to educating employees about cloud technologies across all branches. Therefore, to enhance cloud adoption within SSC effectively, there was a need to increase awareness among all SSC employees about cloud technologies (beyond those teams directly impacted by cloud) and equip them with the necessary skills to adapt their services to cloud.

8.2. Roles and responsibilities needed greater clarity

While roles and responsibilities for delivering cloud services were perceived to be clear and well established among stakeholders (that is, TBS, CCCS, PSPC and SSC), there were gaps identified in procurement. Within SSC, the absence of an internal SSC RACI and uncertainty regarding responsibility allocation for different aspects of cloud services raised questions about the appropriate division of duties. Given the upcoming redesign of the program, it is crucial to align any efforts to clarify roles and responsibilities with the program’s new structure.

8.3. Cloud Services offered access to centralized and standardized services through the portal, but the user experience could be improved

The objective of fostering an enterprise-level approach has been achieved through access to enterprise-level framework agreements, the centralization of the service offering and its standardization. The Cloud Services Portal played an important role, although some would have liked to see it expanded to include more cloud-related services. In this sense, simplifying access to services was also an objective achieved, although with a few caveats. Occasional users found it difficult to acquire a lasting knowledge of the acquisition process through the portal, and those who had to include a business requirement generally found the exercise overly complex.

8.4. Greater awareness and better understanding of the available cloud services and processes were needed

The vision and main objective of SSC’s Cloud Services was to enable intelligent cloud adoption and facilitate the realization of cloud benefits. When it came to smart cloud adoption, the means in place (such as the advisory service and cloud documentation portal) were not known to all users, nor used by the majority. In addition, the optional nature of the program allowed departments and agencies to obtain advice from other providers.

Interviews revealed that several aspects related to the operation of SSC’s Cloud Services were poorly understood, including the competitive obligations related to cloud consumption thresholds, the sharing of certain roles and responsibilities among program stakeholders, and the overall procurement process. This was likely due in part to the complexity of the model used to procure cloud, the dispersal of information across multiple documents, and frequent policy changes on cloud. In fact, one of the most surprising findings of the evaluation was that almost all of the partners interviewed were not aware that SSC’s Cloud Services were optional.

Given that the program has undergone frequent changes over the past few years, and the situation is likely to remain the same for the coming months, SSC should review the way it communicates with departments and agencies to ensure that they have up-to-date information. The literature review discussed in section 6.2 suggests that technological innovation (for example, use of artificial intelligence solutions) could play a role in simplifying communications and services with clients.

The GC objective of facilitating the realization of cloud benefits was well aligned with SSC’s mandate. Indeed, a significant proportion of departments and agencies found that the framework agreements preserved the agility of cloud computing products and services by facilitating rapid access. The responsiveness of the brokering team was noted by many as contributing to this agility. However, some partners and clients felt that the benefits came more from the cloud services themselves than the brokering service. Furthermore, there was a need to balance SSC’s role as a light-touch broker with the objectives of the program. The fact that SSC’s role was limited to its light-touch brokering mandate may have limited the reach of its intervention. The alignment between the cloud program’s objectives and mandate are key to measuring the program’s outcomes.

Given SSC’s available information and data, it is difficult to say to what extent SSC was able to contribute to all of the GC Cloud Adoption Strategy outcomes, namely maximized business value (innovation, scalability, etc.), reduced technical debt or an evolution of the GC’s service-focused culture. Going forward, SSC will need to improve its capacity to evaluate, monitor progress and report to partners and clients. In practical terms, Cloud Services will need to update the logic model and metrics used to measure program performance and to ensure alignment with Delivering Digital Solutions Together and the 2024 cloud and hosting GC strategy. Alignment and coherence among components of the logic models will be key to identifying the most relevant metrics for SSC to tell its story accurately. For instance, if the new program design aims to contain costs, a target and an indicator would be needed to measure performance in this matter. This will also help SSC demonstrate its value.

Appendix A: Definitions

Business value

Is realized by embracing agility, user centricity, automation, sustainability, and evidence-based decision making, enabling organizations to enhance service efficiency and quality.^{Footnote 28}

Cloud Environment

Refers to the infrastructure and resources necessary to support cloud computing services. It includes the hardware, networks, storage and other fundamental elements.^{Footnote 29}

Cloud Service

Refers to infrastructure, platforms or software solutions that are hosted on remote servers and accessible over the internet. Examples include Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS).^{Footnote 30}

Cloud Smart

Is defined by TBS as the strategy by which “the GC will rationalize application portfolios and align to the most appropriate hosting model.”^{Footnote 31}

Digital Transformation

Can refer to anything from IT modernization (for example, cloud computing), to digital optimization, to the invention of new digital business models. The term is widely used in public-sector organizations to refer to modest initiatives such as putting services online or legacy modernization. Thus, the term is more like “digitization” than “digital business transformation.”^{Footnote 32}

(SSC) Enterprise Approach

Is a strategic initiative to centralize IT services across the Canadian government by transitioning departments and agencies from independent IT operations to common IT infrastructure and services. It includes activities to consolidate, standardize, simplify and modernize IT infrastructure and services.^{Footnote 33}

(SSC) Partners and clients

Were the organizations receiving SSC’s services. Partners were federal organizations mandated to receive email, network, data centre, end-user IT and workplace technology device services from SSC. Mandatory clients were mandated to receive a subset of those IT services from SSC. Optional clients had the option to receive IT services from SSC.^{Footnote 34}

Pathfinders

Are organizations that volunteer to pilot new features and services launched by SSC.^{Footnote 35}

Protected B

Refers to a category level in the Government of Canada’s security system. It designates information or assets that require a higher level of protection due to their sensitive nature.^{Footnote 36}

Service-focused culture

Refers to the commitment to understand and meet the needs of customers (and ultimately Canadians) by delivering client-centric cloud services.^{Footnote 37}

Technical debt

Generally refers to the costs that accrue when prioritizing business decisions over technology decisions. It can be seen when existing technology encounters scalability challenges or cannot quickly pivot to meet changing requirements. It can be remediated through continuous activities designed to reduce the scope and liability of the technical debt. This includes activities that reduce the GC’s responsibility for maintaining and replacing technology, such as the use of cloud services.^{Footnote 38}

Vendor lock-in

Occurs when a change in providers would result in an extensive amount of additional time, effort and resources.^{Footnote 39}

Waterfall approach

Is a project management methodology that consists of sequential, phased processes where each phase must be completed before moving to the next, with limited room for iteration or flexibility.^{Footnote 40}

Workload

Refers to the amount of processing that the computer has to perform. This can involve the amount of computational work required to run applications and services, including the processing of data.^{Footnote 41}

Appendix B: Methodology

1. Objective and scope

The evaluation was managed internally by the Office of Audit and Evaluation and covered the period from September 2022 to October 2023. The data collection took place from April to October 2023. The purpose of the evaluation was to inform decision making by providing a neutral assessment of the alignment, effectiveness and efficiency of SSC’s Cloud Services. Based on the initial discussion with program management and scoping interviews, the evaluation originally addressed five questions (Table 1).

Table 1: Evaluation questions

Alignment with the Enterprise Approach	Are GC partner organizations’ current and next steps aligned with SSC’s Enterprise Approach for Cloud? What are the factors supporting and hindering an Enterprise Approach? What are the opportunities to foster the Enterprise Approach for cloud? Are the roles and responsibilities clear and aligned with an Enterprise Approach?
Effectiveness	To what extent does the cloud program support: The GC objectives toward cloud adoption (see section 1) SSC’s objectives toward cloud adoption and ensuring an Enterprise Approach (see section 3) The partner organizations’ objectives toward cloud adoption What are the criteria that will demonstrate the success of the Enterprise Approach?
Efficiency	What are the opportunities to improve efficiency?

The evaluation team reviewed program documentation and collaborated with the Cloud Working Group to develop a notional logic model for this review. The notional logic model identified the desired outcomes and clarified the relationships between program objectives, activities, outputs and intended outcomes. This deliverable appears in Appendix E.

2. Data collection methods

In order to address the evaluation questions outlined above, the evaluation team employed a mixed-methods approach. Utilizing data triangulation, the team cross-referenced the following lines of evidence to identify, analyze and validate findings.

Literature review

In addition to conducting its own literature review on cloud computing and related themes, the evaluation team commissioned 3 literature reviews on the United States’ perspectives on cloud, international perspectives on cloud, and best practices in cloud. The information extracted from the literature reviews was used as a foundation to build a sample for the cross-jurisdictional comparison outlined in the section below.

Document review

The evaluation team synthesized information from 46 documentation sources related to SSC’s Cloud Services, including:

• SSC Governance Board presentation decks (including Executive Oversight Board, Finance, Investment and Internal Management Board, Operations and Services Board)
• Program documents and presentations
• Documents pertaining to GC cloud adoption (including Canada’s Digital Ambition, Canada’s Cloud Adoption Strategy)

This analysis was used to establish knowledge of SSC’s Cloud Services and their evolution throughout time, inform the development of a Cloud Services logic model to support the evaluation, and design the evaluation matrix. To ensure that recommendations remained current and pertinent, the document review was continuously updated until December 2023.

Administrative data analysis

The evaluation team utilized the following data sources to conduct administrative data analysis across fiscal years 2018-2019 to 2022-2023:

• GC InfoBase – Government of Canada (including GC budgets and FTEs)
• Cloud Services Portal (to access the GC cloud service catalogue)
• SSC Intelligent Data 1.2 Dashboard and Enterprise Data Repository (to obtain data on Client Satisfaction Feedback Initiative, Business requirements and Incidents for Cloud Requests)
• Fulfillment Dashboard (which provided data on cloud requests, cloud consumption, brokering fee)
• Actual and Commitments Reports (that is, expenditures)
• Budget Report (which included allocated funding)

This analysis was used to develop insight into partners’ and clients’ historical and current perspective on SSC’s Cloud Services, build awareness of issues and trends related to service delivery metrics and cloud consumption, and understand how program expenditures, revenues and funding have evolved.

Key informant interviews

The evaluation team conducted 60-minute semi-structured interviews to gather information on views and experiences, explanations, and factual information about SSC’s Cloud Services from various stakeholder groups. Interviewees were selected to maximize diversity and representativeness, until data saturation was reached.^{Footnote 42} The following table outlines the interview sample’s composition.

Table 2: Distribution of interviews by respondent category

Interviewee categories	Number of interviews	Number of participants
Internal interviews
SSC employees	28	33
External interviews
GC partner organizations	17	25
GC client organizations	2	2
Other levels of organizations	2	6
GC cloud stakeholders	4	6
Cloud providers	4	7
Total sampling frame	57	78

In the report, the terms “interviews” or “interviewees” denote information gathered both internally (from SSC employees) and externally (from partners, clients and other levels of government). The following table identifies the quantifiers utilized for each category of interviews.

Table 3: Reporting categories of interviews for evaluation report

Quantifier	Internal interviews		External interviews
	N	%	N	%
One	1	<(less than)7%	1	<(less than)7%
Few	2-7	7-25%	2-7	7-25%
Many	8-14	26-50%	8-14	26-50%
Most	>(more than)14	>(more than)50%	>(more than)14	>(more than)50%

Client survey

The evaluation team administered an online survey to capture GC partner and client organizations’ perspectives of SSC’s Cloud Services. The survey collected data from cloud users between July and October 2023.^{Footnote 43} The evaluation team invited all known cloud users with valid emails, most of whom (285) were designated users. The following table outlines the composition of known users by respondent category of designated users and CIOs.

Table 4: Distribution of known users

	User category	Participants invited
Users with valid emails	Designated users only	285
	CIOs only	96
	Designated users and CIOs	12
Total users with valid emails		393
Users without valid emails		19
Total sampling frame		412

As the cloud user community could extend beyond the list of known designated users and CIOs, the invitations provided known cloud users with an open-access link, meaning that participants could share the invitation with anyone that could respond to the survey. To maximize the number of responses, the evaluation team followed the initial invite with 3 reminder emails to respondents. Further, a pop-up invitation was posted on the Cloud Services Portal.

In the end, 99 respondents completed the survey. The number of these surveys completed by known cloud users and the number completed through a shared link are unknown. The response rate could not be calculated, given the unknown population size of cloud users and the snowball method of sampling.

The number of cases available for analysis exceeded 99. A further 32 respondents started the survey, but discontinued before completing it. To maximize the sample size available for analysis, participants who completed a section in its entirety were included in the sample for that section.

Prior to data analysis, the evaluation team considered weighting the data for analysis. The first consideration was the lack of sound population data to compare the survey sample against. Although program data revealed the distribution of known cloud users by factors such as organization size (based on budget), organization type, and cloud consumption rate, the true composition of the user community was unknown.

Even with this limitation, the evaluation team compared proportions of the sampling frame of known cloud users to the resulting survey sample to determine whether the sample was fairly representative. The analysis revealed that similar proportions were present in the sample and the sampling frame of known cloud users. Given the lack of obvious skew, the lack of robust population figures and the modest sample size, the evaluation team decided that weighting was not warranted. An external survey research expert confirmed that, given the limitations, the decision to not weight the data was sound.

Cross-jurisdictional comparison

The team carried out 60-minute semi-structured interviews to capture the experiences, best practices and lessons learned of 11 international (European Union, Belgium, Denmark, Finland, France, Germany, Netherlands, Scotland, Switzerland, United Kingdom and the United States) and 3 provincial (British Columbia, Ontario and Quebec) governments. The analysis was used to compare different jurisdictions to understand their strategies for the management, delivery and use of cloud services and how these different strategies impacted cloud adoption. The following table identifies the quantifiers utilized for each category of international and national interviews.

Table 5: Reporting categories of interviews for evaluation report

Quantifier	Other jurisdictions
	N	%
Few	2	2-25%
Many	3-5	26-50%
Most	>(more than)5	>(more than)50%

3. Limitations

This section highlights crucial limitations to bear in mind while reading this report:

• Ongoing changes to the program made it critical to assess it at a specific moment in time. This snapshot, while useful to the program for lessons learned, must be taken into account when interpreting findings. Recommendations focused on the elements of the program that were likely still relevant in 2024
• While efforts were made to triangulate across multiple lines of evidence, the absence of qualitative and quantitative outcome data for enterprise and service delivery outcomes meant that the results for some questions relied heavily on key informant interviews. To mitigate this, an extensive number of interviews were conducted
• Internal data analysis
  • Financial data on program expenditures, scope of the brokering fee, and program revenues were difficult to obtain. All data presented in this report were validated by both the program and Chief Financial Officer Branch. However, information on other sources of revenue from external business requirements could not be collected at the appropriate disaggregated level to allow for comparison
  • Internal data for the program consists primarily of cloud consumption, brokering fee, finances, and the CSFI questionnaire. Other data (for example, ONYX, BITS and Incident data) were not available for this program as they typically are for other programs at SSC
• Survey
  • The survey findings should be interpreted as indicative rather than conclusive because the generalizability of survey results is unknown. The evaluation was based in part on data from the survey, a tool for which the response rate and degree of representation were not fully known. The list of known users effectively served as a sampling frame, but the number and makeup of the true population of cloud users was not fully known. Thus, the results may not have been fully representative of the user population. For instance, it is possible that the survey sample was skewed towards heavy users, as they may have been more likely to be listed among known users. To mitigate this, the invitations asked participants to forward the invite to those that may have not received an invitation. Still, it was unclear how often that occurred, as was the proportion of the sample that was made up of these respondents

Appendix C: Additional information on SSC’s Cloud Services

1. Policy Environment and related directions

Updates to GC policies and new internal initiatives have shaped the environment in which SSC operates and how cloud services are delivered. Critically, as a result of the 2023 SSC initiative Digital Together, the Cloud Product and Management Services Directorate was lifted out of the Chief Technology Office Branch and merged into a new Hosting Services Branch

2. SSC’s Cloud Services ecosystem

SSC is the program owner of SSC’s Cloud Services, but there are various factors and actors that impact the program’s direction and delivery. Some of these are under SSC’s direct control (including organizational culture and structure, processes and cloud enabling products and services), or are items over which SSC has an influence but is not the only decision maker (including the GC Cloud Strategy, the Directive on the Management of Procurement and the procurement processes, and stakeholder responsibilities). Note that there are other external factors over which SSC has no power or influence (such as cloud provider service offerings and updates, technology evolution and regulations in other countries)

3. Directive on the Management of Procurement

In early 2021, the Directive on the Management of Procurement was designed to optimize the procurement of goods, services and construction, ensuring value and aligning with socio-economic and environmental objectives. It established SSC as a common service provider to federal departments and agencies and ensured that SSC’s procurement of cloud services was aligned with broader governmental objectives and complied with standards of risk management and transparency

4. Procurement Governance Framework

Since mid-2019, the Procurement Governance Framework has applied to GC procurement activities. It ensured that the procurement of goods and services was fair, open and transparent, while providing value for money and demonstrating sound stewardship in support of the delivery of programs and services to Canadians. Under this framework, SSC’s procurement activities focused on delivering the most efficient, effective and highest-value solutions

5. Policy on Service and Digital

In 2020, the Policy on Service and Digital took effect. It articulated how GC departments and agencies should manage service delivery, information and data, information technology, and cyber security in the digital era. It established an enterprise-wide, integrated approach to governance, planning and management. Under the Policy, SSC was responsible for providing related IT services in a consolidated and standardized manner to partners

6. SSC 3.0: An Enterprise Approach

In 2019, SSC launched SSC 3.0, a strategy that defined a whole-of-government approach to consolidate, modernize and standardize GC IT infrastructure. It emphasized the transition from departments operating independently to a common approach

7. Digital Together: The New Hosting Services Branch

In 2023, building on SSC 3.0, Delivering Digital Solutions Together for Canada (Digital Together) continued SSC’s progress in 4 key service areas: connectivity, hosting, digital, and cyber security services. As part of this initiative, a new Hosting Services Branch was created, under which the Cloud Product and Management Services Directorate was placed. Hosting Services aimed to deliver a reliable and sustainable hosting ecosystem where workloads can be transferred from any hosting location without impacting operations, and where enterprise application data can be transferred across GC infrastructure at speed and scale. Promoting a multi-cloud/hybrid model for hosting solutions, SSC’s Hosting Services offered a variety of hosting solutions

8. Updated Application Hosting Strategy

An updated Application Hosting Strategy is set for release in 2024 and will impact all GC departments and agencies governed by the Office of the Chief Information Officer of Canada

Appendix D: Complete roles of SSC’s Cloud Services

Roles and responsibilities of SSC’s Cloud Services

Appendix E: Notional logic model for the evaluation of SSC’s Cloud Services

Long description

Appendix E is a notional logic model for the evaluation of SSC’s Cloud Services program. It illustrates the GC Cloud Strategy objectives, the program main and specific objectives, and the immediate, intermediate and long-term intended outcomes of the program. The diagram uses arrows to show how the objectives and outcomes relate to each other.

GC Cloud Strategy objectives are:

• reduce the accumulated technical debt
• evolve the service-focused culture of the GC
• maximize business value

The GC Cloud Strategy objectives all tie into the Cloud Services Program main objective, which is to enable intelligent GC cloud adoption and facilitate the realization of cloud benefits.

The Cloud Services Program main objective is broken down into two specific objectives:

1. Foster an enterprise approach
2. Accelerate cloud adoption

There are 6 immediate outcomes for SSC’s Cloud Services program. Specific objective 1 is related to the cloud brokering services, and feeds into immediate outcomes 1, 2, 3 and 4. Specific objective 2 is related to cloud advisory services, and feeds into immediate outcomes 3, 4, 5 and 6. The immediate outcomes are:

1. SSC enables partners to use secure and reliable cloud products and services through the GC Cloud Portal
2. Security guardrails contribute to making applications and data more secure
3. Partners gain agility to develop or adapt services to respond to change
4. Partners gain time, expertise and resources by opting for SSC’s centralized services
5. Partners gain knowledge on the cloud via the cloud documentation portal
6. Partners are counseled to make procurement and adoption decisions

There are 3 intermediate outcomes for SSC’s Cloud Services program. Immediate outcomes 1 and 2 feed into intermediate outcome 1. Immediate outcomes 3 and 4 feed into intermediate outcome 2. Immediate outcomes 5 and 6 feed into intermediate outcome 3. The intermediate outcomes are:

1. Cloud adoption is consolidated, modernized and standardized throughout GC departments and agencies
2. Partners accelerate their cloud adoption journey
3. Partners make smart, strategic decisions about cloud adoption.

There are 3 ultimate outcomes for SSC’s Cloud Services program. Intermediate outcome 1 feeds into ultimate outcome 1. Intermediate outcome 2 feeds into ultimate outcome 2. Intermediate outcome 3 feeds into ultimate outcome 3. The 3 ultimate outcomes are:

1. Partners benefit from enterprise-level cloud services that meet a variety of needs
2. Canadian citizens receive access to the digital services they need, when and where they need them, on any device
3. Partners get maximum value from the cloud

List of acronyms

CCCS

Cyber Centre for Cyber Security

CPMSD

Cloud Product Management and Services Directorate

CSE

Communications Security Establishment

CRA

Canada Revenue Agency

CSFI

Customer Satisfaction Feedback Initiative

GC

Government of Canada

IT

Information technology

PSPC

Public Services and Procurement Canada

RACI

Responsible, Accountable, Consulted, and Informed

SaaS

Software as a Service

SSC

Shared Services Canada

TBS

Treasury Board of Canada Secretariat