Evaluation of Mobile Devices and Fixed Lines

This publication is also available online on Shared Services Canada's website. ‌

Permission to Reproduce ‌

Except as otherwise specifically noted, the information in this publication may be reproduced, in part or in whole and by any means, without charge or further permission from Shared Services Canada, provided that due diligence is exercised to ensure the accuracy of the information reproduced is maintained; that the complete title of the publication is produced; that Shared Services Canada is identified as the source institution; and that the reproduction is not represented as an official version of the information reproduced, nor as having been made in affiliation with, or with the endorsement of the Government of Canada. ‌

Commercial reproduction and distribution is prohibited except with written permission from Shared Services Canada. For more information, please contact Shared Services Canada at ssc.information.spc@canada.ca. ‌

© His Majesty the King in Right of Canada, as represented by the Minister responsible for Shared Services Canada, 2023.‌

Shared Services Canada's Evaluation of Mobile Devices and Fixed Lines‌
ISBN 978-0-660-68879-4
Cat. No. P118-28/2024E-PDF‌

Publié aussi en français sous le titre :‌
Services Partagés Canada – Évaluation des services d'appareils mobiles et de téléphones fixes‌
ISBN 978-0-660-68880-0
No. de catalogue P118-28/2024F-PDF‌

Executive summary

Evaluation overview‌

The Office of Audit and Evaluation conducted an evaluation of Mobile Devices and Fixed Lines within the Telecommunications Program. The evaluation assessed the performance of Mobile Devices and Fixed Lines during the period of April 2019 to November 2022. The purpose of this evaluation was to inform decision making and identify potential implications for future operations. The evaluation assessed the alignment, responsiveness, effectiveness and efficiency of Mobile Devices and Fixed Lines. ‌

Key findings‌

Mobile Devices and Fixed Lines were in alignment with enterprise objectives. SSC launched initiatives to modernize outdated telephony technologies, such as the Workplace Communications Project, and to reduce the device-to-user ratio, such as Fixed Line Rationalization. These initiatives supported the Policy on Service and Digital and were aimed at consolidating and standardizing telephony services. These efforts also demonstrated SSC's commitment to good stewardship of Government of Canada resources. ‌

By introducing the Enterprise Service Model (ESM), SSC enhanced its governance to promote enterprise-level priorities. SSC also standardized its reporting on partner progress towards IT provisioning standards as specified in the Policy on Service and Digital. ‌

Despite the many initiatives that aimed at achieving enterprise and stewardship outcomes, there was room for improvement in the effectiveness of Mobile Devices and Fixed Lines in delivering on enterprise objectives. SSC did not achieve its targets for eliminating non-essential fixed lines. At the same time, there was a significant increase in the number of mobile phones provisioned to federal employees. Since March 2021, there have been more mobile phones than public servants. In addition, actual reduction in fixed lines continued to fall behind projected levels, while mobile phones continued to increase at a rate exceeding projections. Based on partner projections for phone line usage contained in the Departmental Plans for Service and Digital, the overconsumption related to Mobile Devices and Fixed Lines will not be solved in the next 3 years. ‌

The evaluation identified a number of factors that hindered SSC's ability to effectively deliver on enterprise objectives. Frequently changing interpretations of the IT provisioning standards set by Treasury Board Secretariat (TBS), ageing infrastructure and complex systems impacted SSC's ability to achieve goals set for modernization and phone line elimination targets. A lack of change management affected partner cooperation and understanding of these important initiatives, which, in turn, created delays and increased costs. External industry pressures such as supply chain and contract issues also hampered SSC's ability to reach its goals. Poor data quality interfered with SSC's ability to make progress towards its objectives as well.‌

From a service delivery perspective, Mobile Devices and Fixed Lines demonstrated a strong commitment to service excellence and made deliberate efforts to meet partners' needs as circumstances changed. However, there was room for improvement in responding to partners' information needs when it came to major change initiatives (e.g., ESM). Partners reported that they needed a clearer picture of the sequencing of service offerings and overarching roadmaps. This impacted partners' and SSC's ability to envision the end-state goal for modernizing and transforming telephony services. ‌

Overall, Mobile Devices and Fixed Lines were effective. The services enabled partner departments and agencies to carry out their mandates by supporting communications, collaboration and enhancing remote and on‑site productivity. However, partners identified a number of challenges in SSC's recent performance in service delivery, including those related to ESM. These challenges impacted partners' confidence in SSC's ability to provide quality services. ‌

The evaluation found that performance measurement throughout SSC was not robust enough to be effective in measuring outcomes. A key management tool for performance measurement is a logic model, which visually depicts the relationship between program activities, outputs and intended outcomes in the short, medium and long term. For telephony services, outcomes were vague and this made it difficult to identify the results that the program was trying to achieve. The logic models also failed to incorporate enterprise outcomes and desired results. Additionally, challenges with the accuracy of the Customer Satisfaction Feedback Initiative (CSFI) reporting appeared to have impacted its use as a reliable indicator to measure performance. This suggests that Telecommunications should consider ways to improve their measurement of and reporting on performance indicators.‌

Accessibility stakeholders identified barriers within Mobile Devices and Fixed Lines that made it difficult for persons with disabilities to access tools that could help them perform their jobs. The Accessible Canada Act (ACA) requires all federal organizations to prepare and publish accessibility plans every 3 years in consultation with persons with disabilities and their organizations. SSC's resulting Accessibility Plan described key barriers to accessibility that aligned with findings from this evaluation. For example, accessibility features were often not enabled on commonly available software and hardware, and accessibility tended to be an afterthought in some areas. Going forward, there is a need for Mobile Devices and Fixed Lines to align service offerings with relevant accessibility requirements as prescribed by the Accessible Canada Act, making services accessible by design and reducing the need for accommodation requests.‌

In terms of efficiency, the evaluation determined that costs per mobile device were competitive. However, overall telephony costs rose from 2020 to 2022 due to increased provisioning (e.g., pandemic and growing public service) and slower than anticipated fixed‑line reduction efforts. SSC made efforts to control rising costs by suspending and cancelling unused mobile phones, piloting softphones and addressing the timeliness of billing information to partners to help address overages. ‌

Of concern, Telecommunications lacked visibility into reliable and detailed cost information for effective decision making and cost management. As SSC is now responsible for complete lifecycle costs for telephony, there is risk of increased cost pressures as Mobile Device and Fixed Line initiatives continue. Without relevant costing information and management tools, programs are unable to effectively model changes and efficiently conduct scenario analyses to predict financial impacts. Financial costing tools will be a key component for ensuring good stewardship over GC financial resources.

SSC also lacked end-to-end processes to manage the lifecycle of Mobile Devices and Fixed Lines. Other telephony processes, especially changes made to processes under ESM, were perceived as cumbersome and inefficient by both partner and SSC interviewees. ‌

Recommendations

1. Develop a standard operating procedure to support the launch of new Telecommunications service offerings or substantive modifications to an existing service offering to partners. This should include defining a communications and change management plan early in the process, and integrating stakeholder engagement and user centered design to help ensure partner telephony needs will be met.
2. Develop a roadmap for all telephony services to support long-term strategic planning. This should include consultations with partners to identify and document their needs and concerns.
3. Develop end-to-end processes for managing telephony services and devices. This should include a comprehensive information management system, documents detailing the roles and responsibilities of SSC and partners and processes for the safe disposal or repurposing of used devices. Partner input should be considered for processes that impact them. These processes should be widely communicated prior to implementation.
4. Review telecommunication services to confirm compliance with accessibility requirements and to ensure services are disability inclusive.
5. Establish and implement a process to develop logic models with specific immediate and intermediate outcomes and associated performance indicators for the Telecommunications Program in the Departmental Results Framework and Performance Information Profiles.

A. Introduction

This report presents the results of an evaluation of Shared Services Canada's (SSC) Mobile Devices and Fixed Lines, which are key components of the Telecommunications Program. In accordance with the Policy on Results, the evaluation assessed alignment with SSC 3.0, the responsiveness (relevance) and effectiveness of service delivery to partners, and the efficiency of Mobile Devices and Fixed Lines. The report is organized into 6 sections: ‌

• Section A provides the description and context of Mobile Devices and Fixed Lines
• Section B describes the evaluation objectives, scope and methodology
• Section C presents the key findings related to achieving enterprise objectives, including the alignment of program activities with SSC objectives, the effectiveness of the program in achieving desired enterprise and stewardship outcomes, and the factors that affected achieving these outcomes
• Section D presents the key findings related to service delivery, including responsiveness to changing client needs and the achievement of service delivery outcomes for partners
• Section E presents the key findings related to efficiency
• Section F summarizes the study's conclusions and provides recommendations

1. Program description

1.1 The Telecommunications Program‌

During the period under review from April 2019 to November 2022, the Telecommunications Program provided voice, teleconferencing, contact centre services and the supporting telephony devices to enable communications in the Government of Canada (GC) workplace. The program was responsible for ensuring service continuity while modernizing telecommunication services for partner organizations and agencies, enabling mobilization of the workforce.‌

Under the 2016 Treasury Board Policy on Results, every department in the GC has established a Program Inventory that identifies all of the department's programs and describes how resources are organized to contribute to the department's Core Responsibilities and Results. According to SSC's 2022/23 Program Inventory, the Telecommunications Program included the following program components:‌

1. Mobile Devices and Fixed-Lines
2. Conferencing Services
3. Videoconferencing
4. Telecommunications Multi-Service Support
5. Contact Centre Infrastructure Services
6. Toll-Free Voice

The Telecommunications Program was delivered by the Digital Services Branch (DSB). Program components were managed under 2 directorates within the branch. The Telecommunications Directorate managed Mobile Devices and Fixed Lines. The Contact Centre and Conferencing Services Directorate managed the other 5 components.‌

1.2 Mobile Devices and Fixed Lines program component‌

Mobile Devices and Fixed Lines addressed the communication requirements of a wide variety of GC employees, which ranged from desk workers to highly mobile field workers. Telephony services were essential for employees who spent the majority of their day speaking on the phone and those who relied on the data-intensive capabilities of feature-rich mobile devices to do their work. These services were integral to the effective delivery of GC services to clients.‌

Specifically, the services provided under the Mobile Devices and Fixed-Line phones program component included: 

Mobile device services

Mobile device services provisioned devices, accessories, and service plans for SSC partners and clients. Mobile device networks were managed by vendors, while service requests were managed by SSC. The mobile device services available were: ‌

• Basic cell phones and voice and text plans
• Smartphones and voice, text and data plans
• Cellular-enabled tablets, Wi-Fi hubs and data-only plans
• Specialized devices and push-to-talk
• Subscriber Identity Modules (SIM) cards

Mobile device services were dependent on a vital enabling service, Enterprise Mobile Device Management (EMDM), which allowed mobile devices to securely connect to GC assets. SSC staff operated the EMDM service, which managed applications on smartphones. EMDM services included:‌

• Synchronization of enterprise email and calendar to phones
• Deployment of commercial and custom applications to phones
• Secure mobile access to GC extranet and partner intranets

Fixed line services

Fixed line services provisioned landline systems, services and devices using a combination of vendor-managed and SSC-managed services. Services included:‌

• Voice over Internet Protocol (VoIP)
• Central exchange (Centrex)
• Private branch exchange (PBX)
• Long distance telephone services (LDTS)
• Calling cards

1.3 Mobile Devices and Fixed Lines spending

Mobile Devices and Fixed Lines was the largest component of the Telecommunications Program. Total expenditures for Mobile Devices and Fixed Lines have increased from $250.7M in 2020/21 to $339.6M in 2022/23.‌

1.4 Program performance measurement‌

In accordance with the Treasury Board Policy on Results, departments across the GC developed Performance Information Profiles (PIPs) that identified the performance information for each program in their Program Inventory. At SSC, each branch has been responsible for establishing, implementing and maintaining their PIPs. ‌

A logic model is an essential part of the PIP. Logic models visually depict the relationship between program activities, outputs and intended outcomes in the short, medium and long term. They also include the impact of a program and how a program links to related departmental core responsibilities.‌

Program logic models help determine what results a program should measure. A clear logic model should illustrate the purpose and context of the program. It also serves as guidance for developing meaningful evaluation questions and an effective evaluation methodology to assess program success in delivering the desired results. ‌

The Telecommunications Program logic model from the 2022/23 PIP defined the following immediate, intermediate and ultimate outcomes for Mobile Devices and Fixed Lines:‌

• Immediate outcome: Voice services are well-managed and effectively procured
• Intermediate outcome: Voice services are available and enable collaboration and productivity
• Ultimate outcome: Government departments and agencies receive modern and reliable network services

Interestingly, these outcomes focused exclusively on service delivery outcomes to partners. The logic model did not include any outcomes that focused on achieving enterprise objectives or reflected the stewardship role that Shared Services Canada plays. ‌

To assist with the identification of outcomes for the evaluation, a notional logic model was developed. The notional logic model was for evaluation purposes only and did not cover all components of the Telecommunications Program. The Notional Logic Model for the evaluation of Mobile Devices and Fixed Lines is in Appendix C.

2. Program context

2.1 Background‌

SSC was created in August 2011 to bring together IT infrastructure resources from 42 departments and agencies (SSC partners). The scale, scope and complexity of this type of merger was unparalleled in the GC. ‌

One of SSC's responsibilities included managing and consolidating telephony infrastructure from SSC partners at sites across Canada. While SSC received some telephony employees from partners during its creation, some corporate memory loss of telephony infrastructure still occurred. Due to this corporate memory loss, information management for telephony infrastructure and services became disjointed. Over time, the number of phone lines also grew. Legacy networks became outdated, with many reaching end-of-support or end-of-life status. ‌

To help address these challenges, SSC launched the Cost-Effective Telephone Services initiative in 2016, specifying that public servants should have a single telephony device. As part of that initiative, SSC shifted to a "mobile first" approach and encouraged partners to "cut the cord" to reduce the number of fixed-line phones. Accordingly, the number of mobile phone requests increased. ‌

The Shared Services Canada Act was amended in 2017 to enable some partners to manage some services, including mobile devices, in-house. While this eased the pressure on SSC, it also resulted in a loss of control and knowledge of mobile device services. ‌

2.2 Evolving context during the evaluation period‌

During the evaluation period from April 2019 to November 2022, Mobile Devices and Fixed Lines underwent a period of substantial change that impacted how the services were delivered, funded and governed. This included the launch of SSC 3.0, the new Policy on Service and Digital, and the introduction of the Enterprise Service Model.‌

2.2.1 SSC 3.0: An Enterprise Approach

In 2019, SSC launched its new strategy for service delivery. SSC 3.0 defined a whole‑of‑government approach to consolidate, modernize and standardize the GC IT infrastructure. It emphasized the transition from independent operations to common or enterprise approach. ‌

Consistent with the enterprise approach, SSC's mandate for telecommunications was to transition the GC to a common, shared telecommunications infrastructure. As the common IT service provider, SSC was responsible for delivering modern and reliable telephony services. As the steward of GC resources, SSC was also responsible for delivering these services with a view to reduce costs, centralize their administration and rationalize service delivery. This was expected to achieve greater efficiencies, minimize risks, improve security and improve service quality.‌

2.2.2 Policy on Service and Digital

In April 2020, the new Treasury Board Policy on Service and Digital was published. The Policy articulated how GC organizations would manage service delivery, information and data, information technology and cyber security in the digital era. It established an enterprise-wide, integrated approach to governance, planning and management. Overall, the Policy advanced the delivery of services and the effectiveness of government operations. It also supported the Minister for Digital Government's mandate to lead Canada's digital transition. The management of these functions would be guided by the principles and practices outlined in GC Digital Standards. Under the Policy, SSC was responsible for providing related IT services in a consolidated and standardized manner to partners. ‌

Of significance, the Standard on Information Technology Provisions (the Standard) in the Directive on Service and Digital provided concrete policy support for SSC's efforts to consolidate, modernize and standardize IT infrastructure. The Standard provided specifications for the provision of information technology to departments. According to the Standard, partners' consumption of IT services and provisions was subject to defined entitlement standards, accompanied by enhanced departmental service consumption reporting. ‌

2.2.3 Enterprise Service Model

The Enterprise Service Model (ESM) was announced as a budget decision in 2021 and implemented in April 2022. ESM worked within the framework provided by the Directive on Service and Digital to consolidate SSC services, including Mobile Devices and Fixed Lines. It represented a whole-of-government enterprise shift, where every organization would use common IT infrastructure and each user would have common tools. It was expected that ESM would allow for faster turnarounds, enhanced collaboration, increased reliability and reduced risk.‌

Specifically, ESM was intended to: ‌

• provide stable funding to enable longer term planning
• support the management of demand while ensuring service capacity
• be simple and cost effective to deliver
• be transparent to partners
• support an enterprise approach to service delivery

Prior to ESM, funding for SSC services was overly complex, inconsistent and costly to administer. Funding consisted of a patchwork of cost recovery and appropriations. In some cases, SSC billed partners for some or all of the service, while in other cases SSC was responsible for paying for the full service. The introduction of ESM eliminated the need to bill each partner for enterprise services. It was intended to provide SSC with better control to streamline common services and promote good stewardship of GC resources.‌

In order to fund ESM, a one-time appropriation was made from partner budgets in April 2022. Appropriations for most services, including Fixed Lines, were estimated from partners' actual usage of services in fiscal year 2019/20. For Mobile Devices, the appropriation was estimated based on each partner's usage in September 2020. This funding was intended to cover services now provided by SSC, including cost and volume protection. Going forward, cost recovery would only be used if warranted for a limited number of services, such as specific one-time business requests.‌

While ESM created benefits, the new model was not without risks. Rising costs and changing partner demands created a new need for SSC to carefully monitor available funding versus costs to ensure fiscal responsibility.

B. Methodology

This section describes the objective and scope of the evaluation, the specific evaluation issues and questions that were addressed, data collection methods and data limitations.‌

1. Objective and scope

The purpose of the evaluation was to inform decision making by providing a neutral assessment of the alignment, responsiveness, effectiveness and efficiency of Mobile Devices and Fixed Lines. After the completion of the initial evaluation report, additional data collection was conducted to assess SSC's performance in achieving its enterprise and stewardship objectives regarding Mobile Devices and Fixed Lines. This report presents findings on the responsiveness, effectiveness and efficiency of SSC's delivery of telephony services. It also presents findings from the additional data collection, which focused on SSC's enterprise alignment and stewardship role. ‌

The evaluation addressed the following key questions:‌

1. To what extent have Mobile Devices and Fixed Lines been aligned to SSC 3.0?
2. How responsive have Mobile Devices and Fixed Lines been to changing client needs?
3. How effective have Mobile Devices and Fixed Lines been in achieving the intended outcomes?
4. What are the opportunities to improve efficiency?

The evaluation covered the period from April 2019 to November 2022. Data collection began in January 2022 and initially concluded in November 2022. In response to arising requirements, additional data collection took place from December 2022 to March 2023 and again from April to May 2023. The additional data collection focused on the degree to which the program had aligned its actions to the enterprise approach. ‌

The evaluation was managed and conducted internally by the Office of Audit and Evaluation in accordance with the Treasury Board Policy on Results. The evaluation team was supported by external expertise to conduct the literature review and the partner survey. ‌

2. Data collection methods‌

The evaluation used a mixed-method approach. Multiple lines of evidence, along with the triangulation of data, were used where possible to address all evaluation questions. Specifically, the evaluation used literature reviews, document reviews (including SSC Service Reviews), key informant interviews, observations, administrative data analysis and a client survey. There were a number of limitations, including the difficulty of disentangling the beginning of ESM from systemic issues, data gaps for line counts and the reality that partner perspectives findings reflect their perspectives at a specific point in time. Given that ESM was implemented in April 2022, it was too soon to determine whether ESM had achieved its intended impacts.‌

More information on the lines of evidence and the limitations appear in Appendix B.‌

C. Findings related to enterprise objectives

This section describes evaluation findings related to enterprise and stewardship objectives. It looks at the extent to which Mobile Devices and Fixed Lines sought to foster and support the enterprise approach. It also assesses whether the services were successful in achieving the desired enterprise outcomes. Finally, it identifies the factors that hindered the achievement of these outcomes.‌

1. Alignment with enterprise objectives

The evaluation examined the alignment of Mobile Devices and Fixed Lines with the enterprise approach. Overall, the evaluation found that SSC's actions were aligned with the enterprise approach and supported the Policy on Service and Digital.

1.1 SSC supported Treasury Board policy updates‌

The Policy on Service and Digital introduced the Standard on Information Technology Provisions in April 2020. The Standard specified a limit of one cellular device along with the lowest-tier cellular plan for GC IT Users and Call Centre Users. ‌

During the implementation process of the Standard, technology and user requirements changed; telephony services also evolved and converged. Based on lessons learned from ESM and the early implementation stage of the Standard, SSC continuously revisited targets for modernization. SSC also collaborated with Treasury Board Secretariat and the GC Enterprise Architecture Review Board (GC EARB) to propose updates to the Standard.‌

Telecommunication's latest interpretation of the telephony provisioning standard was proposed in February 2023, specifying a one‑user‑to‑one‑telephony‑device ratio, whereby a user would be allocated one telephony device that was the most conducive for their workplace technology requirement. This interpretation had evolved from the previously proposed target of 1.2 devices per full-time employee (FTE). At the same time, interpretations of the Standard were subject to the needs of partners, recognizing that some job functions would have unique requirements. ‌

1.2 SSC enhanced its enterprise governance and reporting‌

Prior to the introduction of ESM, governance and funding was structured around individual partners with differing preferences that did not always align with enterprise-level priorities. One of the expected results of ESM was stronger enterprise governance and priority setting. This, in turn, could allow SSC to deliver on enterprise objectives and meet partner enterprise IT needs within existing reference levels. ‌

SSC standardized its reporting on partner progress towards IT provisioning standards using Departmental Plans for Service and Digital (DPSDs). SSC Client Executives and the Telecommunications Directorate collaborated with partners to build achievable plans and record these plans in DPSDs. These reports helped to forecast the rate at which departments would become compliant with the Standard established in Treasury Board policy. ‌

Compliance status would then be reported to the GC Enterprise Architecture Review Board. Service consumption beyond entitlement standards required special approval from GC EARB. When exceptions were submitted for approval, partners were required to provide a rationale for the exceptions, and SSC was required to provide the associated cost and schedule implications. This approval process helped to ensure that exceptions to policy standards were sufficiently justified. ‌

1.3 SSC modernized outdated telephony technologies‌

Traditional landline telephony systems were based on legacy technologies that were no longer reliable or cost effective. At the time of data collection, many partners still relied on legacy fixed‑line infrastructure. Some of these networks had reached end-of-support or end-of-life status. Telecommunications vendors informed SSC they would soon stop offering some legacy services. It was becoming increasingly difficult to locate necessary expertise and equipment to address service outages related to these systems.‌

In alignment with the Treasury Board standard for service provisioning, SSC focused on transitioning users from traditional desktop phones to mobile phones that supported modern GC work environments. For users who continued to require fixed lines (for example, users who had accommodation requirements or worked in security zones), legacy voice services were modernized and transitioned to enterprise Voice over Internet Protocol (VoIP) services. VoIP services were administered centrally from enterprise data centres, helped reduce support costs and improved security and service delivery. ‌

To mitigate challenges with VoIP implementation at sites that did not have local area networks (LANs) or upgraded facilities, SSC implemented ‘VoIP-basic' solutions instead of full VoIP services. VoIP-basic delivered modern voice services on legacy LANs. Minimal upgrades (e.g., power, heating ventilation and cooling), which were a partner's responsibility, were required to support these solutions, thereby alleviating funding issues for partners. VoIP-basic solutions mitigated risks associated with end-of-life legacy voice infrastructure and offered a reliable service to meet business requirements for many years. ‌

Modernization work was also ongoing under the Workplace Communication Services (WCS) Project. The WCS Project modernized legacy phone lines for the Department of National Defence and a small number of other government departments. Specifically, WCS aimed to modernize 170K legacy Private Branch Exchange(PBX) to enterprise VoIP services. As of December 2022, 27.2K PBX lines had been migrated to VoIP.‌

SSC also modernized legacy infrastructure through Real Property Projects. These projects ensured modern enterprise solutions were the default for new infrastructure requirements and existing workplace fit-ups. SSC worked with partners to determine the optimal telephony solution for their voice service requirements. Enterprise VoIP and cellular solutions were recommended for real property projects whenever possible.‌

For mobile phones, SSC developed a mobile device refresh plan to ensure that GC mobile devices remained up to date. Under this plan, GC mobile phones followed a "3 + 1" year refresh model and provided users with a new device after 3 years. This approach balanced cost efficiency with GC security requirements and the functionality needs of users. ‌

1.4 SSC implemented initiatives to reduce device-to-user ratio‌

In alignment with Treasury Board Standards for service provisioning, SSC implemented initiatives to eliminate superfluous and unused telephone lines. ‌

SSC introduced the Fixed Line Rationalization (FLR) initiative to eliminate non-essential fixed line phones. The goal of FLR was to improve the efficiency of telephony services across government and ensure judicious use of public funds. SSC also implemented the zero usage initiative to reduce the number of unused mobile phone plans across the GC. A more detailed discussion of these initiatives is contained in section E – Findings Related to Efficiency.‌

2. Effectiveness – Enterprise and stewardship outcomes

The evaluation examined SSC’s effectiveness in achieving enterprise and stewardship outcomes. Overall, SSC made some progress towards achieving enterprise and stewardship outcomes, but there was room for improvement. The evaluation also identified a number of factors that hindered SSC’s progress towards achieving enterprise and stewardship outcomes.

2.1 SSC made some progress in eliminating phone lines, but the pace was slow‌

In February 2020, SSC developed an integrated service strategy for Mobile Devices and Fixed Lines. According to this strategy, SSC aimed to eliminate superfluous fixed lines at an average rate of at least 10% per year. ‌

In March 2020, shortly before the introduction of the Policy on Service and Digital, there were 556,120 fixed lines across partner organizations (source: Enterprise Data Repository, extracted June 5, 2023). The number of fixed lines was reduced to 548,650 in March 2021, and further to 514,003 in December 2022, representing an average annual reduction rate of 2.8%. This was significantly below the 10% target.‌

Unfortunately, SSC did not achieve its target elimination rate for fixed lines. ‌

SSC worked closely with the Office of the Chief Information Officer (OCIO) of the GC to manage the number of GC phone lines. This was mainly managed according to the provisioning standards established in the Directive on Service and Digital, which took effect in April 2020. ‌

In the GC, a "workpoint" was any space where employees could perform their work. Each workpoint was equipped with furnishings and digital tools that supported a variety of tasks and varying degrees of interaction or concentration. ‌

The Directive on Digital and Service defined 9 IT user and workpoint profiles. The most common profile was the "GC IT User." A GC IT user was defined as a general IT user equipped to work at and away from a workpoint, outside GC offices, or in telework or remote work situations. Under the Standard on Information Technology Provisions, GC IT Users were entitled to one cellular device and one lowest-tier cellular plan. Exceptions could be made for secure or classified information or communications, where a second cellular device and second cellular plan could be provided.‌

From March 31, 2020, to March 31, 2023, the number of indeterminate employees, term employees, casuals and students in the federal public service (i.e., the core public administration and separate agencies) grew from 300,450 to 357,247, representing an 18.9% increase.‌

During the same period, the number of mobile phones provided to this population increased at a considerably higher rate. Specifically, the number of mobile phones (including suspended devices) increased from 232,307 in March 2020 to 403,358 in March 2023. These 171,051 new phones represented a 74% increase over 3 years. ‌

While some public servants (such as classified users) may have had valid reasons for having additional devices, there were more mobile phones than public servants. Specifically, the ratio of mobile phones (including suspended devices) to public servants increased from 0.77 in March 2020 to 1.13 by March 2023.‌

Long description

The graph shows that the number of Mobile Phones have exceeded the number of public servants since March 2021.

Year‌	Number of Public Servants‌	Number of Mobile Phones‌
March 2020‌	300,450‌	232,307‌
March 2021‌	319,601‌	321,193‌
March 2022‌	335,957‌	347,140‌
March 2023‌	357,247‌	403,358‌

Sources: TBS Population of the federal public service and Canada (Population of the federal public service - Canada.ca); data from SSC (Enterprise Data Repository (EDR), accessed May 2023).

2.2 While SSC made progress towards achieving GC standards, projected levels of phone usage could not be sustained

Together, FLR and the enhanced reporting required under ESM were working to bring SSC partners closer to the standard of one telephony device per user profile. The DPSDs) contained partner projections of their phone needs for the next 3 years. SSC compiled this information into reports to track partner progress on reducing over consumption. According to the 2022-2023 Consumption by Partner report, partners projected they would increase their mobile devices by 33,843 and decrease their fixed lines by 130,185 in fiscal year 2022/23. Comparing these DPSD projections with the actual change in partner phone line usage demonstrates that the number of fixed lines did not decrease as much as projected by partners. At the same time, it shows that the number of mobile devices increased more than partner projections.

Long description

The graph shows that in fiscal year 2022-23, the number of mobile devices increased more than projected by SSC partners and the number of fixed lines did not decrease as much as projected by partners.

Type of Phone‌	Projected Change‌	Actual Change‌
Mobile Devices‌	+ 33,843‌	+ 56,218‌
Fixed Lines‌	- 130,185‌	- 54,976‌

Sources: Based on data from SSC (2022-23 Consumption by Partner report; EDR, accessed May 2023).

As reported in the Quarter 4 2022-2023 Consumption by Partner report, as of April 2023, there were 840,855 total phone lines (excluding suspended mobile devices), but only 680,875 user profiles and workpoints. This represented an overconsumption of 159,980 phone lines. Furthermore, partner projections for their expected 3-year demand for Mobile Devices and Fixed Lines was 797,179 phone lines. These projections indicate that there would still be an overconsumption of 116,304 phone lines in fiscal year 2025/26. According to the DPSD report, "in aggregate, across the GC, SSC does not have sufficient appropriations to sustain actuals at these levels." ‌

Based on these projections from the 2022-2023 DPSDs, there was no evidence that this overconsumption issue would be resolved in the next 3 years.‌

Long description

The graph shows that both actual phone usage among SSC partners in April 2023 and partners’ projected phone usage in fiscal year 2025/26 exceeded SSC’s target phone usage. This illustrates that as of April 2023, there was no evidence that the issue of phone overconsumption in the GC would be resolved in the next three years.

Target Phone Usage‌	Projected Phone Usage in FY 2025-26‌	Actual Phone Usage in April 2023 (excluding suspended mobile devices)‌
680,875‌	797,179‌	840,855‌

Source: Based on data from SSC (Q4 2022-23 Consumption by Partner report)

3. Analysis of hindering factors

The evaluation identified 7 factors that have hindered SSC’s effectiveness in achieving desired enterprise and stewardship outcomes :

Long description

There were seven factors that hindered SSC’s effectiveness in achieving outcomes:

1. Lack of clarity on Enterprise objectives, standards, and timelines
2. Continued use of legacy voice services by partners
3. Insufficient change management
4. Slow partner response to changes
5. Poor data quality
6. Industry trends and supply chain issues
7. Lack of ability to enforce standards

3.1 There was a lack of clarity on enterprise objectives, standards and timeline

The evaluation team observed some confusion over the enterprise approach and its objectives among employees in the Telecommunications Program and elsewhere in SSC. It was unclear the extent to which this lack of clarity impacted SSC's ability to make progress on its objectives. ‌

There was also confusion over the Telecommunication Program's various interpretations of the IT provisioning standards set by TBS. From 2019/20 to 2022/23, there were 5 different interpretations of the IT provisioning standards. The initial interpretation in June 2021 set a ratio of 1.4 telephony devices per FTE, shifting to 1.2 devices per employee in February 2022. The proposed interpretation in early 2023 was 1 telephony device per user profile plus non-human lines (e.g., elevators, security lines, etc.) as required. While most of these interpretations were not officially ratified, partners were aware of some of them. This presented a moving target to partners, which may have made it difficult for some partners to align their telecommunication plans to SSC's interpretation of the IT provisioning standards.‌

According to the program document review, the Telecommunications Program identified timelines to achieve telephony service consumption targets within some strategy updates and proposals. These timelines had been communicated in an inconsistent way. For example, in fiscal year 2019/20, the telecommunications integrated service strategy specified a target to reduce Centrex and PBX lines by 10% per quarter. In fiscal year 2021/22, Telecommunications identified a target to achieve 1.2 phone lines per FTE by fiscal year 2024/25. This inconsistency in communicating timelines, coupled with the different interpretations of the IT provisioning standards, could have made it difficult for SSC to define and achieve phone line elimination targets.‌

It is important to note that by fiscal year 2022/23, timelines to comply with the IT provisioning standards set by TBS were developed by partners during the digital planning process. ‌

3.2 The continued use of legacy voice services by partners created challenges for SSC for modernization and consolidation‌

The legacy environment that many partners still relied upon for voice services caused complications and delays for modernization. In particular, ageing IT infrastructure and complex systems were problematic.‌

One of SSC's goals was to consolidate partner voice services. This was a significant and onerous task that was hampered by a complex patchwork of different partner systems and approaches. At the time of data collection, SSC was still working through vendor invoices to identify and rectify billing errors and over-billing that originated before SSC's creation.‌

Furthermore, according to internal interviewees, telephony architecture was too complex. This complex infrastructure could introduce several points of failure when migrating partners from legacy to modern services. This could create delays when something went wrong, since SSC would have to "follow the breadcrumbs" to determine what happened. There were also a lot of standalone fixed‑line sites that required time-consuming individual visits for software patches and updates. SSC was working on consolidating these sites to a central core with a goal to centralize administration. ‌

Ageing partner infrastructure also hampered efforts to modernize telephony infrastructure. According to interviewees, some of the back-end technology partners relied upon was over 30 years old. In many cases, this legacy infrastructure had to be replaced before a partner could be onboarded to modern service solutions. Sometimes it could cost millions of dollars to replace or upgrade legacy infrastructure.‌

3.3 There was insufficient organizational change management ‌

Change management is a collective term for approaches to prepare, support and help individuals, teams and organizations in making organizational changes. Telephony services had undergone a period of substantial changes before and during the evaluation. These changes significantly impacted how the services were delivered, funded and regulated. These changes continuously brought risks and opportunities to both SSC and its partners. In order to successfully manage the changes that affected partners, SSC needed an effective change management strategy. ‌

SSC had a number of internal supporting functions available such as Change Management, Behavioural Insights, and Service Design. These functions were designed to provide certain in‑house expertise to help programs design and manage changes to services. External expertise was also available to support SSC in navigating critical organizational change processes. ‌

The evaluation did not find evidence that SSC had leveraged appropriate change management expertise. As a result, many issues occurred, which impacted SSC's ability to deliver effective telephony services to partners and meet enterprise objectives.‌

Section D takes a closer look at SSC's responsiveness to partners' information needs.‌

3.4 Some partners were slow to respond to changing requirements‌

Due to the lack of organizational change management, partners did not always have sufficient understanding of the reasons behind the changes. For example, since partners had not paid for fixed line services since SSC's creation in 2011, they did not have visibility on the cost of non‑essential phone lines. Furthermore, many partners did not understand that the initial appropriation amounts to SSC in 2011 had been eroded due to increased costs and demands for the services. In the words of one SSC interviewee, "In general, one of the challenges right now is that partners don't appreciate that the function of SSC is to save the GC money. They don't see the savings, and often the changes SSC makes are seen as increasing costs on their end."‌

The Fixed Lines Service Review and interviews for this evaluation identified 3 additional factors that contributed to the slow response by partners. Some partners had competing priorities and lacked financial incentives to complete the work. Others lacked staff to complete the work while maintaining essential operations. Finally, some partners did not believe they had a reliable replacement for fixed lines. Due to these factors, the planned reduction in fixed lines did not occur on schedule. As of October 2022, 43% of partners had not completed the line validations needed before SSC could eliminate a fixed line.‌

Section D takes a closer look at the sequencing of service offerings and roadmaps. ‌

3.5 Poor data quality was a barrier to making progress towards enterprise objectives‌

Poor data quality was another barrier for SSC's progress towards achieving its goals. This included both data that SSC was responsible for maintaining and other data sources that SSC relied upon for decision making.‌

There was a need for SSC to improve the information management of its assets within Mobile Devices and Fixed Lines. It was well‑known that prior to FLR line validation exercises, SSC did not have complete knowledge of which partners owned which fixed lines and where the fixed lines were located. There were 2 main reasons for this. First, SSC received incomplete information about partner telephony assets upon its creation in 2011. Second, there was no clear process to inform SSC of changes to fixed‑line ownership when partners stopped using offices. As a result, SSC had to rely upon partners to validate which lines they owned and where they were located. This was a lengthy process that had to be completed before partners could inform SSC which fixed lines could safely be disconnected. According to internal interviewees, SSC also did not have complete knowledge of some mobile devices. This related mainly to partners who had taken over provisioning of their own mobile devices, as per Bill C-44 from 2017. In some cases, differences between SSC's and partners' records for mobile devices had to be reconciled when reference levels for ESM were calculated.‌

There were also challenges with external data that SSC relied upon to set telephony targets for partners. Initially, partner phone line elimination targets were calculated using FTE data from each partner. However, it was later identified that partner FTE data did not provide an accurate representation of the number of public servants who required a phone line. This led to revisions to the proposed target ratio of phones to public servants.‌

3.6 Some industry trends impacted SSC's modernization efforts‌

There were some factors that SSC had limited control over. These external pressures included influences from the telecommunications industry and supply chain issues.‌

Some of SSC's Fixed Lines and most of its Mobile Devices (EMDM services as the only exception) were managed by vendors. This meant that vendors and industry trends impacted the services. For example, vendors were phasing out their Centrex service offerings, but some partners still relied on Centrex lines. SSC was able to negotiate a contract extension for Centrex services but at a significant increase in cost. ‌

Another way in which vendors impacted SSC's progress towards its goals was through their performance in relation to contracts. For example, the Workplace Communication Services project to modernize fixed lines was contracted out to an external vendor. The contract was deeply affected by severe delays that finally resulted in SSC redefining the scope of the work.‌

During the evaluation, global supply chain issues also impacted SSC. There was a chipset shortage that had wide-reaching impacts across the IT industry. This resulted in order backlogs and sometimes impacted which phones partners could obtain. Internal interviewees identified that careful planning and bulk phone orders could help offset supply chain issues.‌

3.7 SSC lacked the ability to enforce standards

While entitlement standards for telephony were an important policy tool to assist SSC to rationalize usage of phone lines, they were not enough on their own to meet line reduction targets for fiscal year 2022/23. According to internal interviewees, SSC lacked the ability to force partners to cut fixed lines. SSC also lacked the ability to say "no" when partners requested mobile devices, regardless of whether a request exceeded a partner's DPSD projections or entitlement standards. While this was an appropriate approach to ensure that partners had unimpeded access to needed services, it meant that SSC was not applying enforcement mechanisms to ensure partner compliance with standards. As a result, SSC relied heavily on partner cooperation to bring Mobile Device and Fixed Line usage in line with GC standards. ‌

D. Findings related to service delivery

This section describes evaluation findings related to responsiveness and effectiveness for service delivery outcomes. Relevance is the extent to which a program addresses a demonstrable need. Responsiveness is a dimension of relevance and it focuses on the extent to which a program’s objectives and design respond to beneficiary or partner needs and continue to do so as circumstances change.

1. Responsiveness to changing client needs

The evaluation examined how responsive Mobile Devices and Fixed Lines had been to changing client needs. Overall, the evaluation found that while SSC was committed to providing client-centric services, there was room for improvement in meeting partners’ needs.

1.1 While staff were committed to service excellence, SSC was not successful in responding to partners’ information needs

During interviews and in CSFI comments, partners consistently reported that Telecommunications staff were committed to providing good service and were great to work with. When surveyed in Q3 2022/23, 71% of partners agreed that SSC worked quickly to resolve serious issues with voice services. This was further supported by CSFI results from October 2020 to April 2022, which indicated that Mobile Devices and Fixed Lines met or exceeded target customer satisfaction scores.

Quotes from interviewees from partner organizations:

However, SSC was not seen as successful in responding to partners' information needs on a number of initiatives. ‌

According to external interviews, while the introduction of ESM was communicated widely, communication was not timely and did not trickle down efficiently to all relevant levels of staff. Interviewees reported that only high-level details were shared before ESM started. Only 33% of partner respondents believed that SSC had provided enough details about changes to voice services under ESM for them to make the changes in time for ESM's implementation. ‌

At the time of the interviews in summer 2022, partners reported that many operational-level details were still missing 3 months after ESM was implemented. For example, partners lacked details on processes for device evergreening, processes for reconfiguring phones when users changed departments and whether there was a cap on ordering new mobile devices. According to interviewees, late communication about ESM impeded partners' abilities to complete their budgets and IT plans. Partners had been given details on ESM in February 2022 and this was after they had completed their planning for 2022/23. According to survey results in Q3 2022/23, only 39% of partners believed that SSC provided enough details for them to plan accordingly.‌

SSC's challenges in responding to partners' information needs led to perceptions that SSC did not understand partners' operational needs. In the survey in Q3 2022/23, only 39% of partners believed that SSC understood their operational requirements for voice services.‌

Long description

Results from four questions in the survey administered by the evaluation team in quarter three of fiscal year 2022-23.‌

• 71 percent of partners agreed that SSC worked quickly to resolve serious issues with voice services.
• Only 33 percent of partners reported that SSC provided enough details about changes to voice services under ESM for them to make the changes in time for ESM's implementation.
• Only 39 percent of partners believed that SSC provided enough details about ESM for them to plan accordingly.
• Only 39 percent of partners believed that SSC understood their operational requirements for voice services.

1.2 The sequencing of service offerings needed improvement

In summer 2022, some partners indicated there was misalignment between voice service standards, SSC initiatives and service offerings and partner needs. In particular, the sequencing of service offerings was problematic.‌

The Directive on Service and Digital stipulated that most public servants were entitled to a mobile phone for their work. However, at the time, many GC buildings had insufficient cellular signals. The upgrades required to boost cell phone signals were expensive and not within the budget of some partners. ‌

Despite pressure from SSC to eliminate fixed lines, some partners believed that they did not have a viable alternative. These partners believed SSC was attempting to shut down one capability before another was enabled. This perception impacted partner progress on FLR. According to the survey in Q3 2022/23, while 78% of partners believed FLR was aligned with their needs, only 57% believed that SSC provided a realistic deadline for their organization to complete FLR activities.‌

While SSC had created roadmaps for some initiatives, such as FLR, the softphone pilot and EMDM activities, multiple lines of evidence suggested that an overarching plan for all telephony was missing. According to interviewees, this made it difficult for partners to visualize the future of telephony services. This was echoed in the survey in Q3 2022/23. Only 47% of survey respondents said that SSC had shared a clear roadmap for the future direction of voice services.‌

During the evaluation period, there had been several recent changes to telephony services with more changes planned. At the time of data collection, large departments reported that they were struggling to keep up with the changes. A comprehensive voice services roadmap would better enable partners and SSC to envision the end-state goal for the transformation of telephony services and plan accordingly.‌

As of June 2023, there was a Telephony roadmap that was being planned to be shared with partners.‌

2. Effectiveness – Service delivery outcomes

In evaluation, the core issue of effectiveness focuses on the impacts of a program and the extent to which the program is achieving its expected outcomes. Outcomes are the changes or consequences attributable to program outputs.‌

In terms of effectiveness in service delivery to partners, the evaluation considered telephony devices and services, client service and the quality of those services. It also examined how effectively services were monitored and measured through internal tools such as performance indicators and the CSFI. ‌

Overall, Mobile Devices and Fixed Lines were effective. The program supported the ability of partners to carry out their mandates and serve Canadians; however, partners expressed concerns about the deteriorating quality of service. This was partially due to cumbersome SSC processes. The evaluation also identified some organization-wide challenges at SSC with performance measurement.‌

2.1 Mobile Devices and Fixed Lines enabled partners to carry out their mandates

During the evaluation period, federal employees relied on voice services, such as Mobile Devices and Fixed Lines, to enable their work in serving Canadian citizens. Multiple lines of evidence showed that SSC’s telephony services were effective. When interviewed, partners independently and frequently indicated that SSC’s telephony solutions were critical services that enabled their workforce to stay connected to each other and to Canadians wherever they were. SSC telephony services enabled collaboration and increased the productivity of public servants.

SSC’s Departmental Results Report for 2021/22 indicated that the cellular network was available more than 99.5% of the time. Additionally, in the Q3 2022/23 survey, 69% of partners reported that SSC’s mobile phone services supported their organization’s delivery on its mandate. For fixed lines this was 84%.

Long description

Results from two questions in the survey administered by the evaluation team in quarter three of fiscal year 2022-23.

• Overall, 69 percent of partners reported that SSC’s mobile phone services adequately supported their organization’s delivery on its mandate.
• For fixed lines, 84 percent of partners reported that the services adequately supported their organization’s delivery on its mandate.

Interviewees provided many specific examples of how Mobile Devices and Fixed Lines supported, enabled and enhanced effective tasks. They helped safety organizations protect national security and uphold the safety of Canadians, and helped science organizations facilitate the collection of data. EMDM-enabled applications permitted public servants to enter data in real-time during inspections or other field work, saving time and boosting productivity.

The tethering feature on smartphones allowed public servants to connect a GC tablet or laptop to the Internet so they could continue working when they did not have reliable Wi-Fi. Cell phones kept public servants connected to their teams and emergency services at events and while doing field work. SSC’s deployment of cell phones during the COVID-19 pandemic enabled public servants across Canada to continue working and serving Canadians from home.

2.2 Partners expressed concerns about the deteriorating quality of service in 2022‌

While SSC's commitment to supporting partners was notable, interviewees in summer 2022 indicated that SSC had suffered damage to its reputation during the evaluation period due to recent performance. This reputational damage led some partners to believe that SSC was not capable of providing the level of service they required.‌

Partners' internal telephony teams reported that they were often blamed by end-users for poor service and indicated it was now outside their control to help these users directly. In Q3 of 2022/23, only 6% of survey respondents working in partner departments believed that SSC's performance in fulfilling voice service requests under ESM had a positive impact on their reputation with their clients. ‌

Furthermore, only 44% of partners surveyed in Q3 2022/23 were confident about SSC's ability to meet their voice services needs under ESM. As a result of this lack of confidence, some interviewees and survey respondents expressed a desire to take back management of some or all aspects of Mobile Devices.‌

According to partner interviewees, there were several challenges that impacted Mobile Devices and Fixed Lines that contributed to a decrease in partner confidence in SSC's ability to deliver voice services. These included:‌

• ESM challenges, including its rushed implementation and lack of clarity
• asset and information management gaps
• insufficient consultation with partners prior to implementing service changes
• training gaps and software errors related to Onyx (the IT Service Management tool)
• lengthy delays in fulfilling partner requests
• cumbersome processes (including service desk processes and EMDM processes)
• unmet partner needs related to international telephony and service sequencing

These service issues likely contributed to the drop in Mobile Devices' CSFI rating. Prior to the implementation of ESM, the average rating peaked at 4.12 out of 5. By October 2022 this average rating was 3.02 out of 5, below the 3.60 target rating. It was also not clear why a 3.60 rating would be sufficient. This is lower than what many organizations set as their target. ‌

Long description

Line graph demonstrating partner ratings of mobile devices and fixed lines services in the Customer Satisfaction Feedback Initiative questionnaire on services from October 2018 to April 2023. In general, mobile devices and fixed lines met or exceeded SSC’s target rating of 3.60 (out of 5.0) since October 2020. However, ESM-related challenges impacted partner ratings for mobile devices in October 2022.

Date‌	Mobile Devices Score‌	Fixed Lines Score‌
October 2018‌	2.74‌	3.74‌
April 2019‌	3.32‌	3.66‌
October 2019‌	3.52‌	3.58‌
October 2020‌	3.90‌	3.60‌
April 2021‌	4.02‌	3.81‌
October 2021‌	4.12‌	3.93‌
April 2022‌	3.79‌	3.86‌
October 2022‌	3.02‌	3.88‌
April 2023‌	3.72‌	3.95‌

Source: Based on data from SSC (CSFI Service Questionnaire results)

2.3 There were organization-wide challenges at SSC with performance measurement

According to the Treasury Board Policy on Results, outputs are direct products and services that come from the activities of an organization, program or initiative. Outputs are usually within the control of the organization itself. By contrast, program outcomes are changes or consequences that are attributable to the direct products or services of the program. ‌

The 2021/22 PIP performance indicators for Mobile Devices and Fixed Lines were: ‌

• Indicator 1: Percent of time the fixed-line voice infrastructure is available
• Indicator 2: Customer satisfaction with telecommunications services

These indicators were not robust enough to sufficiently measure identified outcomes. Of specific concern, infrastructure availability directly stems from the activity of the program. This would be an output, not an outcome.‌

From a conceptual level, using customer satisfaction to measure outcomes arising from SSC performance for partner organizations was problematic. Specifically, customer satisfaction was a proxy for measuring the achievement of outcomes. Customer satisfaction could be vague, based on perceptions and affected by expectations. It could reflect satisfaction with outputs (for example, what was within the control of the program). This meant that customer satisfaction did not directly capture the real impacts of excellent or poor SSC performance and the ways in which SSC supported partners in delivering on their mandates.‌

There was room for improvement in the identification of immediate service delivery outcomes for the Telecommunications Program. The outcomes developed in the 2021/22 Performance Information Profile (PIP) for Mobile Devices and Fixed Lines were:‌

• Immediate outcome: Voice services are well-managed and effectively procured
• Intermediate outcome: Voice services are available and enable collaboration and productivity

These outcomes were vague and this made it difficult to identify the results that the program was trying to achieve or assess the bang-for-buck for investments in these services. ‌

It should be noted these types of performance measurement challenges were observed across all of SSC's PIPs, and the Departmental Results Framework (DRF). ‌

To address this gap, the evaluation team conducted a survey and asked partners whether Mobile Devices and Fixed Lines had adequately supported their organization's ability to deliver on its mandate. ‌

Looking forward, partners provided several suggestions on the best ways to measure the business impact of SSC performance from a service delivery perspective. The top 3 suggestions that partners felt were the best way to measure the impacts of poor or excellent performance by SSC on end users were:‌

• track and report on time to fulfill requests from partners and resolve issues
• obtain feedback from end-users about their experience, including whether the service meets user needs
• track and report on the accuracy of request fulfillments from partners

Other suggestions for performance measurement from partners included:‌

• determine ease of use, availability, and quality of self-support information and technical support
• develop and track SSC performance on service-level standards and report results
• compare the cost of GC services to public offerings
• track the number of times a client must follow up on a request
• track the number of escalations required for requests or issues to be resolved

When it came to how partners measured the performance of their own teams, 94% of partner respondents tracked request fulfillment time within their department or agency, while 71% tracked client satisfaction and 34% tracked provisioning errors.‌

2.4 Partners and SSC employees raised issues with the CSFI that had implications for it's accuracy in measuring SSC performance‌

The purpose of the CSFI was to provide quarterly measurements of partner satisfaction with SSC services in general, as well as with specific services, such as Telecommunications. Several challenges with this measurement tool surfaced independently and consistently during partner interviews, and this was further supported by evaluation survey results. ‌

In Q3 2022/23, only 33% of respondents believed that the mechanisms SSC used to receive feedback from partner organizations enabled clients to clearly communicate concerns. These challenges had implications for the CSFI's use as a robust and reliable indicator for measuring service line performance.‌

Some partners had multiple telephony teams that all provided responses to the CSFI. Interviewees reported that their responses were averaged before submission to SSC. This created situations where, if one area received excellent service and another received poor service within the same partner organization, this level of detail was lost. This impacted the accuracy of SSC's understanding of partner satisfaction. ‌

Additionally, some partners submitted their CSFI responses with an awareness of their audience, which impacted their ratings and comments. Of specific concern, some partners reported that they had moderated their critical ratings and comments on the CSFI due to a perception that their Client Executive's performance bonus was linked to CSFI survey results. Partners viewed Client Executives as their champions within SSC for resolving their issues with service delivery. ‌

This finding suggests that Telecommunications should consider ways to improve their measurement of and reporting on performance indicators.‌

2.5 There were barriers within Mobile Devices and Fixed Lines services that made it difficult for Persons with Disabilities to access tools that could help them perform their jobs‌

The Accessible Canada Act is a federal law that came into force in 2019, which aims to make Canada barrier-free by 2040. Part of the focus of the ACA is to identify, remove and prevent barriers in employment for persons with disabilities. It requires all federal organizations to prepare and publish accessibility plans in consultation with persons with disabilities and their organizations.‌

SSC 3.0 recognizes that "Federal employees need modern digital tools that are accessible by design; enhance their productivity and allow them to collaborate across departments to deliver the essential services Canadians rely on."‌

At the time of data collection, there was no document outlining service line responsibilities to ensure information and communication technology was accessible by design. Inclusion requirements were often not proactively considered when telephony systems or services were procured. For example, when VoIP devices were launched, they were not evaluated for accessibility. This meant that public servants who required accessibility features on fixed lines had to use legacy devices and technology. This was not in line with the ACA's goals.

Adaptive technology in smartphones could assist the GC in meeting its accessibility goals; however, SSC's configuration of smartphones made accessibility features difficult or impossible to use. This resulted in a lack of options for public servants with disabilities. In the words of one interviewee, when SSC made changes to its configuration of smartphones, "my phone went from being partially usable to not usable at all. It is not the device but the GC layer, policies, lack of software available in the workspace." According to the interviewee, smartphones configured by SSC were difficult to even unlock for some persons with disabilities. Other examples of accessibility features that were difficult to obtain at the time of data collection included Siri (which is a critical enabling feature for people with visual impairments) and document readers. In contrast, some departments that did not use SSC's Mobile Device services did not have the same challenges to make accessibility options available on smartphones when needed.‌

The lack of accessibility options in GC Mobile Devices and Fixed Lines at the time of data collection contributed to a need for public servants with disabilities to submit accommodations requests to SSC’s Accessibility, Accommodation and Adaptive Computer Technology program. While accommodation requests were a critical service, ad-hoc solutions were often time consuming and costly compared to an accessible-by-design approach, and they often could not fully meet the needs of public servants with disabilities. Building accessibility features into Mobile Devices and Fixed Lines services at the onset would reduce the need for accommodations requests, saving resources and promoting inclusion.

It is important to note that SSC released an Accessibility Plan in December 2022 that had been developed in consultation with persons with disabilities. The Accessibility Plan described key barriers to accessibility that aligned with findings from this evaluation. For example, accessibility features were often not enabled on commonly available software and hardware, and accessibility tended to be an afterthought in some areas. The Plan also developed goals to address accessibility shortcomings. Going forward, there is a need for Mobile Devices and Fixed Lines to build accessibility into their services at the onset to align with accessibility goals.

E. Findings related to efficiency

The evaluation examined whether the services were delivered in a cost-effective and timely manner, and whether there were opportunities to improve operational efficiency. ‌

Overall, SSC's telephony expenses had risen since the start of the pandemic because of significant increases in the number of mobile phones provided to public servants. During the evaluation period, the program tried to reduce costs by reducing the number of fixed lines. It also made considerable efforts to contain costs by identifying and cancelling unused mobile phones; introducing softphones; and providing billing information to partners in a more timely manner. ‌

Under the new ESM, SSC carried all of the risk for any uncontrolled spending. To ensure the successful delivery of the new strategic roadmaps for services and identify opportunities for efficiency improvements, SSC needed improved costing information for decision making. Specifically, there was a need for complete cost information for services (including benchmarks and disaggregated data), better lifecycle cost information, and new scenario planning tools. ‌

Furthermore, the evaluation also found a number of areas in telephony processes where efficiency could be improved. ‌

1. SSC’s telephony costs were increasing

While it is difficult to find relevant benchmarks for commercially confidential information, a review of industry trends suggested that SSC had negotiated excellent rates for voice, text and data cell phone plans. GC rates were significantly lower than retail rates. SSC also took advantage of bulk purchases of devices to receive price discounts.

Despite these competitive rates, SSC expenditures on telephony services increased from $196.7 million in 2019/20 to $264.3 million in 2021/22. This represented a 34% increase over 2 years. When SSC assumed responsibility for all Mobile Devices and Fixed Lines costs under ESM in 2022/23, total expenditures jumped to $339.6 million, representing a 73% increase since 2019/20. This was 12% higher than the initial budget allocation of $303.5 million.

Long description

Graph illustrating expenditures for fixed lines, mobile devices, and their combined total for fiscal years 2019-20 to 2021-22. Total combined expenditures for mobile devices and fixed lines increased by 34 percent since 2019-20.

Fiscal Year‌	Mobile Devices Expenditures‌	Fixed Lines Expenditures‌	Total Expenditures‌
2019-20‌	$59.8 M‌	$136.9 M‌	$196.7 M‌
2020-21‌	$118.4 M‌	$132.3 M‌	$250.7 M‌
2021-22‌	$120.1 M‌	$144.2 M‌	$264.3 M‌

Source: Based on data from SSC (obtained from the Telecommunications FMA)

The main driver for increased expenditures was the rise in the number of devices provided to public servants across the GC.

In addition to mobile devices, the GC will always need to maintain some traditional fixed lines for dial-up modems, security alarms, elevators, business continuity plans, secure phone and fax lines, published numbers, and work sites in remote locations without access to other networks.

At the same time, there was widespread agreement that many of the 514,003 fixed lines (as of December 2022) across the GC could be eliminated. Accordingly, the Fixed Line Rationalization (FLR) initiative was launched in 2020 to reduce superfluous fixed lines. According to the Q4 2022/23 Consumption Report, the number of fixed lines decreased by 12.3% (from 585,015 to 513,058) from Q1 to Q4 in 2022/23. The consumption report indicated that the target was to decrease the total number of phone lines to 680,875, not broken down by separate fixed line and mobile targets. Given that there were 128,804 fixed-line workpoints and 21,881 high security workpoints identified in the report, the target number of fixed lines could be estimated at 150,685.

Long description

Graph illustrating the number of fixed lines, mobile devices (excluding suspended devices), and total phones in each quarter of fiscal year 2022-23. The number of fixed lines decreased by 12.3 percent from quarter one to quarter four in fiscal year 2022-23. This in turn has helped reduce the total number of phone lines.

Quarter‌	Total Fixed Lines‌	Total Mobile Devices (excluding suspended devices)‌	Total Phones‌
Q1 2022-23‌	585,015‌	338,871‌	923,886‌
Q2 2022-23‌	535,244‌	349,392‌	884,636‌
Q3 2022-23‌	523,255‌	357,782‌	881,037‌
Q4 2022-23‌	513,058‌	327,797‌	840,855‌

Source: Based on data from SSC (Q4 2022-23 Consumption by Partner report)

2. Mobile devices and fixed lines worked to contain costs

During the pandemic, mobile phones were provisioned for public servants to enable them to contact their teams and provide services to Canadians while working from home. Since then, the adoption of MS Teams has decreased many public servants’ reliance on mobile phones. This was especially true for public servants who worked from a desk in a specific location, as many of them used MS Teams as their primary mode of communication. These public servants made minimal use of their mobile phones. The number of mobile devices that had not been used for 3 or more months increased by 65% between March 2020 and March 2023. At the same time, the service plan fees for these unused phones cost hundreds of thousands of dollars per month, totaling $5.9 million in 2021/22 and $3 million in 2022/23. The average monthly cost of unused phone service plans was $253,832 in 2022/23.

Long description

Graph illustrating that the number of mobile devices that had not been used for three or more months increased by 65 percent between March 2020 and March 2023.

Year‌	Number of mobile devices unused for three months or more‌
March 2020‌	57,750‌
March 2021‌	97,008‌
March 2022‌	91,021‌
March 2023‌	95,160‌

Source: Based on data from SSC (Telecommunications Internal Services)

To help address the issue of unused mobile phones, SSC implemented the zero usage initiative. This initiative created mandatory suspension measures for mobile phones that had 3 months or more of zero usage, and cancellation measures for phones that had 6 months or more of zero usage. In consideration of the needs of partners, SSC made allowances for emergency devices that had to always be available. In the June 2022 Zero Usage Report, SSC identified 91,237 mobile service accounts to be suspended and ultimately disconnected. However, delays in finalizing cancellations occurred as partners required extensions for their reviews of these devices. Based on partner feedback, Telecommunications had planned to cancel 52,000 zero-use mobile phones by spring 2023.‌

In fiscal year 2022/23, an update to the mobile device refresh plan was proposed to shift from 100% flagship smartphone devices to a combination of flagship devices (for 20% of users) and second-tier devices (for 80% of users). Flagship devices referred to the most advanced smartphones offered, while second-tier devices were the best-value smartphones that met GC specifications. This proposal was made in order to support ESM and reduce the predicted Telecommunications budget deficit.‌

To support the decommissioning of more expensive legacy technologies while still meeting the majority of user requirements, SSC was in the process of adding softphones to its service catalogue. The 2022 Service Review for Mobile Devices noted that softphones could be more cost effective than mobile phones. Softphones are software that allow users to make and receive phone calls over the Internet using a laptop, tablet or smartphone. At the conclusion of this evaluation, softphones were being designated as the preferred way forward by the Telecommunications Program. A softphone pilot was in progress at the time of the evaluation. As of March 2023, 16 departments had participated in the softphone pathfinder initiative, using a total of 2,700 softphone lines. The softphone deployment plan included a phased approach to replace mobile phones as the preferred telecommunications service where possible. Telecommunications planned to obtain service authorization for the softphone service in August 2023.‌

This solutions-based approach is believed by SSC to be consistent with Treasury Board policy. Under the Standard on Information Technology Provisions, each fixed-line workpoint is to have 1 fixed telephony/network connection. It is not entirely clear whether the intent of the Standard was to provide each end user with an individual fixed line connection (e.g., landline phone) or whether a fixed line into each building could be shared among multiple users through new network technologies. ‌

Finally, the Telecommunications Internal Services team had noticeably improved the timeliness of reporting since spring 2020 to help in the management of costs. At one point, there was a 3‑month delay in the consolidation and release of billing information for mobile devices to partners. In interviews, several partners independently raised this issue and reported that these kinds of delays eroded their ability to take action on unnecessary data overages in a timely manner. In one example, a partner had a mobile data device in the field that had been hacked. The partner was charged about $220,000 in data overages before they were made aware of the issue. By December 2022, SSC was releasing reports in about 2 weeks after receiving the invoice from the vendor. ‌

3. There was a need for more granular cost information and more robust cost accounting tools for decision making

As part of the efficiency analysis, the evaluation reviewed the information and tools that were available for cost management. The expectation was that all service costs would be available, separately identifiable and complete. It was also anticipated that complete per-device annual costs and lifecycle costs would be available for appropriation calculations and management decision making.‌

Some of the elements necessary for effective cost management decision making were present. At a foundational level, it was clear that SSC had a chart of accounts with cost centres and that all SSC expenses were coded against these cost centres. Information on the number of service plans for Mobile Devices and Fixed Lines was also available and easily accessible. ‌

The main tool that SSC has utilized to establish recovery agreements with partners for business requests has been the Enterprise Price Estimation Tool (ePET). The ePET met departmental needs for these kinds of estimates. Specifically, ePET estimates included the main direct costs (such as service plan costs and device costs), some indirect costs (such as shipping costs and select salaries) and an SSC corporate overhead charge, which was updated every 2 years. Moreover, ePET added a 20% contingency fee, which ensured that SSC always received sufficient funds to cover expenses for business requests and never ran deficits.‌

While the ePET may have been very effective for business recovery estimates and protected the department against project cost overruns, it was not an appropriate tool for cost management decision making in a world where SSC would now have to absorb all telephony costs. Of concern, indirect costs (such as device disposal, repair, detailed salaries, etc.) were not included in the ePET. In addition, the ePET was not sensitive to differences in costs for devices, data plans or for SSC to deliver services in different regions.‌

The need for the department to have more complete information and good cost management tools was demonstrated by the calculation of ESM appropriations in 2022. ESM simplified how partners paid for some enterprise services (including telephony) and it eliminated the need for cost recovery agreements. The shift to an appropriated model was intended to stabilize service delivery to enable longer-term planning.‌

This new funding model required a one-time transfer of relevant funds from partners to SSC for telephony. However, poor data quality impacted the appropriation calculations. SSC did not have a way to accurately project growth in mobile phone use by SSC partners, as the appropriation was calculated in September 2020, but ESM was not implemented until April 2022. From the time of the calculation to the date of implementation, the number of mobile phones (including suspended devices) across the GC increased by 22%, between September 2020 and March 2022 (from 284,005 to 347,140 phones). Because of this, SSC was not properly funded for mobile phone services for the first fiscal year of ESM. Furthermore, the number of mobile phones continued to grow after the implementation of ESM, reaching 403,358 (including suspended devices) in March 2023, which has created further cost pressures. ‌

Long description

Graph illustrating that mobile phones (including suspended devices) among SSC partners have steadily increased from September 2020 to March 2023.

Year	Number of mobile phones (including suspended devices)‌
September 2020‌	284,005‌
March 2021‌	321,193‌
September 2021‌	327,605‌
March 2022‌	347,140‌
September 2022‌	369,435‌
March 2023‌	403,358‌

Source: Based on data from SSC (EDR, accessed May 2023)

Without an assessment of all related costs, SSC did not have complete and reliable financial information for financial planning and accurate ESM appropriations. When the appropriations were calculated, they did not reflect partners' true telephony costs. Some partners reported that they had too much reallocated from their budgets while others reported that they had too little. There was no break-even analysis for changes in volume or breakdown by partner.‌

Of concern, there were missing indirect costs that would typically be considered when determining the full cost for telephony. These indirect costs would include the cost for replacement devices, which would be estimated based on the number of active phones. There would be disposal costs and warehousing costs given that the space could be used for other purposes. In addition, there would be costs for help desks and support for installations and updates to devices. Finally, full costing would also take into consideration the impact of volume adjustments and changes in the number of telephony plans and devices.‌

Given that not all Mobile Devices and Fixed Lines costs were analyzed, a complete accounting of the costs of incremental changes in the provisioning the services could not be provided to the evaluation team. For example, the complete year-over-year cost of a single additional smartphone issued in the GC could not be provided. This lack of comprehensive costing also affected the transparency of per device costs. Without a complete internal cost report, SSC and its partners lacked knowledge of the incremental cost to SSC of provisioning additional mobile devices for a partner.‌

Looking forward, SSC will be acting as the steward of GC funds for IT infrastructure and the department will be implementing a series of strategic roadmaps to modernize services. To improve efficiency, detailed costs for year-over-year comparisons will need to be tracked.‌

Better costing information would allow the department to understand where and why changes in expenses are occurring. Programs and services could take action to improve efficiency and avoid unnecessary cost burdens. SSC could conduct reviews to identify value-added and non‑value-added activities. Finally, it would foster a better understanding of costs across the full lifecycle of new services. The total costs over the full lifecycle may differ significantly from the original acquisition price. In some cases, it may be cheaper to replace aging equipment than to pay the ongoing maintenance and support costs. ‌

In this context, there is a need for a more robust cost accounting tool to establish baselines, pinpoint changes in costs, and assist in management decision making for the rollout of strategic roadmaps. Specifically, an improved costing tool would:‌

• improve the ease and accuracy of conducting sensitivity analyses (for example, determining the financial impact of changing device types for mobile devices)
• provide a method to increase the granularity of cost details to assist in determining where cost efficiencies can be made, above and beyond simply decreasing the number of phones (for example, determine how much is being spent on repairs per device or per partner)
• increase agility for planning (for example, shifting to a "softphone preferred" model)
• enhance SSC's ability to respond to unexpected events (for example, a global pandemic)
• allow for lessons learned as expenses are tracked year over year

4. There was room for improvement in the management of the telephony lifecycle

During the evaluation period, telephony infrastructure within the GC was complex. Departmental barriers led to isolated services and inconsistent user experiences. Due to this complexity, it could be challenging for the program to manage telephony devices and systems through their lifecycle. An improved ability to efficiently track devices and services could contribute to better cost control, stronger security and improved stewardship of GC resources.‌

During the evaluation period, there was uncertainty among partners and some Telecommunications staff about roles and responsibilities for mobile devices. This created inefficiencies in the lifecycle management of mobile phones. For instance, there was confusion around processes for mobile phone reconfigurations when public servants switched departments. There was also a lack of clarity on responsibilities for broken devices. When end‑users had an issue with their phone that could not be addressed through their departmental service desk, a replacement device was issued. Many end-users did not receive instructions on what to do with their broken device so users held on to them indefinitely. ‌

As of October 2022, SSC's warehouse contained large numbers of used mobile phones awaiting disposal or recycling, including old BlackBerry devices and flip phones. As mobile phones became more widely used across the GC, partners sent more and more devices to SSC and the stockpile of used phones "got out of control," according to SSC staff. The warehouse team developed processes and was hiring staff to take care of the backlog. At the time of the evaluation, they hoped to start the disposal process soon but anticipated it would take time to get through the substantial backlog. ‌

It is unknown how many phones were in the warehouse and it was not clear whether the used phones contained classified information. Of additional concern, some Android devices posed a safety risk to the warehouse and staff as those devices contained batteries that could swell without consistent recharging. ‌

Newer phones that were not past their useful life were sometimes sent to SSC's warehouse. Partners assumed SSC was reusing the phones, but phones that had been sent back to the warehouse were not being reused at the time of the evaluation. While the warehouse team had plans to repurpose newer phones, as well as a long-term goal to repair phones with small issues for reuse, this part of the telephony lifecycle required more attention.‌

5. Telephony processes were perceived as inefficient

SSC processes were another major pain point for interviewees. Both SSC and partner interviewees cited that cumbersome or ill-defined processes impacted efficiency. ‌

According to SSC interviewees, some SSC processes were slow, complex or ill‑defined. Service desk interactions were a big pain point and lacked flow charts or documents that effectively explained processes to partners. Some interviewees indicated it was confusing for partners to identify which part of SSC to engage for different requests. The IT Service Management (ITSM) software itself was also seen as problematic. For example, if a partner did not include some required information, their entire request could be rejected and they would have to start their ticket over. ‌

Business requests (BR), especially EMDM requests for new applications, were viewed as lengthy, complex and expensive processes that were not communicated well. SSC interviewees indicated that it could take months and thousands of dollars to complete simple tasks, like adding a commercial application to GC smartphones. In the words of one interviewee, "Sometimes I feel like we're adding costs for very simple tasks." To request a mobile application, partners had to undergo both the BR process (6 steps) and a mobile application vetting process (5 steps). Information on mobile application requests was not available on the Serving Government website. Instead, partners had to visit 2 different GCpedia pages to learn how to request applications. ‌

Partner interviewees believed that the changes made to processes and services under ESM were not well thought out. SSC and partner organizations did not have time to determine the effects ESM would have on the workflows within partner and SSC telecommunications teams. As a result, partner interviewees perceived processes under ESM to be even less efficient than before. Routine processes, such as mobile phone provisioning, password resets and SIM card activations, were disrupted. This resulted in delays of 4 or 5 weeks for requests that had previously been completed in under 5 business days. Partner actions to alleviate these challenges, such as circumventing processes, in turn created inefficiencies for SSC. One interviewee commented, "SSC doesn't have an enterprise mindset yet. They will pick something to change without thinking about how it will affect the services in other areas."‌

Before ESM, some partners could order their devices directly from the vendor. Under ESM, all partners had to request devices through SSC. This added extra steps to the provisioning process. In some cases, partners saw their workflow to provision a mobile device go from 3 or 4 steps pre-ESM to up to 6 or 8 steps post-ESM. As a result, many partners viewed SSC as a "middleman" that negatively impacted their efficiency. Furthermore, despite SSC's responsibility for mobile device provisioning and management, partners reported they had just as much or more work related to these devices as before, which strained their resources. Despite having decreased telephony budgets, some partners had to hire additional employees to help track and follow up on trouble tickets, coordinate device evergreening or collect timely data on mobile device usage. In total, only 9% of survey respondents in Q3 2022/23 reported that their organization required fewer employees to manage telephony under ESM. Partner interviewees pointed out that while they needed to maintain or increase their telephony employees under ESM, SSC also needed to hire additional staff, increasing the total number of telephony employees throughout the GC. Partners questioned whether resources were being wasted under ESM and voiced a need for an alternative solution that balanced partner needs with SSC responsibilities.‌

F. Conclusions and recommendations

Major conclusions reached during the evaluation are summarized. A set of recommendations is presented to improve the ability of Telecommunications to meet its objectives for Mobile Devices and Fixed Lines.

1. Conclusions

Mobile Devices and Fixed Lines were in alignment with SSC 3.0. SSC provided telephony services in a consolidated and standardized manner, demonstrating alignment with enterprise objectives and a commitment to good stewardship of GC resources.‌

While SSC implemented many initiatives in support of the service provisioning standards specified by the Policy on Service and Digital, SSC did not achieve its targets for reducing the device-to-user ratio. There are now more mobile phones issued than federal employees. In addition, SSC did not have sufficient appropriations to sustain projected levels of phone usage for the next 3 years. ‌

SSC's lack of progress in achieving these objectives could have been attributed to a number of complex factors. These included the Telecommunications Program's frequently changing interpretation of the IT provisioning standards set by TBS, aging infrastructure, a lack of change management, external industry pressures, and poor data quality. ‌

From a service delivery perspective, SSC demonstrated a commitment to service excellence and to making deliberate efforts to meet partners' needs. However, there was room for improvement in SSC's responsiveness to partners' changing needs, especially information needs. ‌

Overall, SSC was effective in enabling partners to carry out their mandates by supporting and enhancing remote and on‑site productivity. However, partners expressed concerns about the deteriorating quality of service. Additionally, there were organization-wide challenges at SSC with performance measurement. Challenges with the accuracy of Customer Satisfaction Feedback Initiative reporting also appeared to have impacted its use as a reliable performance measurement indicator. ‌

In terms of efficiency, the evaluation determined that costs per mobile device were competitive. However, overall telephony costs rose from 2020 to 2022. To contain costs, SSC made various efforts, including suspending and cancelling unused mobile phones, piloting softphones and addressing the timeliness of billing information to partners to help address overages. ‌

There was a need for more granular cost information and more robust cost accounting tools for decision making. There was also room for improvement in the management of the telephony lifecycle and in streamlining telephony services. ‌

2. Recommendations

The conclusions of the evaluation led to the following recommendations:‌

1. Develop a standard operating procedure to support the launch of new Telecommunications service offerings or substantive modifications to an existing service offering to partners. This should include defining a communications and change management plan early in the process, and integrating stakeholder engagement and user-centered design to help ensure partner telephony needs will be met.
2. Develop a roadmap for all telephony services to support long-term strategic planning. This should include consultations with partners to identify and document their needs and concerns.
3. Develop end-to-end processes for managing telephony services and devices. This should include a comprehensive information management system, documents detailing the roles and responsibilities of SSC and partners, and processes for the safe disposal or repurposing of used devices. Partner input should be considered for processes that impact them. These processes should be widely communicated prior to implementation.
4. Review telecommunication services to confirm compliance with accessibility requirements and to ensure services are disability inclusive.
5. Establish and implement a process to develop logic models with specific immediate and intermediate outcomes and associated performance indicators for the Telecommunications Program in the Departmental Results Framework and Performance Information Profiles.

Appendix A: Service review recommendations

Service review recommendations - mobile devices‌

• Communicate the strategy to support the next procurement cycle for EMDM to keep relevant stakeholder groups informed.
• Develop a strategy to align mobile device services with other services that support remote work and mobility. The service review noted that smartphones may duplicate laptop capabilities, and softphones may be a more cost-effective solution than mobile phones.
• Update the target operating model to reflect changes in the mobile strategy in support of remote work, the telephony strategy and the adoption of ESM.
• Re-evaluate the enterprise IT standard policy, including the mandatory requirement to provide all GC employees with smartphones, and consider and evaluate softphones and Bring Your Own Device models.
• Develop the means to track, report and manage the costs associated with mobile device services at the enterprise level, including a resolution mechanism for unplanned costs resulting from partner over usage.

Service review recommendations – fixed lines‌

• Review existing and potential complementary telephony standards to provide clear guidance to partners on requirements associated with fixed line migration and the broader telecommunications environment.
• Review existing contracts and identify fixed end dates to help set clear expectations with partners and mitigate migration strategy risks.
• Evaluate opportunities to accelerate migration by providing partners with financial incentives. In combination with telephony standards and sourcing, financial incentives could strengthen SSC's approach to migration.
• Enhance the capacity and ability of fixed line services and partners to complete GC profiles and related discovery, inventory and mapping activities by providing supplemental resource capacity to fixed line services.

Appendix B: Data collection methods and limitations

Evaluation findings were based on 6 lines of evidence. There were also 5 limitations that affected the findings in this report.‌

1. Literature reviews‌

Two literature reviews were conducted. The first examined best practices in Mobile Devices and Fixed Lines in both the private and public sectors. This analysis was used to gauge how SSC's provision of telephony services compared with industry norms.‌

The second literature review was a comparative analysis of Mobile Devices and Fixed Lines practices from governments around the world. The analysis highlighted different approaches to telephony by other governments.‌

2. Document review‌

A document review was conducted to gain an understanding of Mobile Devices and Fixed Lines in the GC. The document review also provided insights into the priorities and objectives of SSC in general, and Telecommunications in particular. The service reviews were the first documents consulted. Other key documents included:‌

• corporate planning and strategy documents
• Departmental Results Framework
• program documentation
• policy documents

3. Key informant interviews‌

Semi-structured interviews were conducted to gather in-depth qualitative information for evaluation purposes. Information gathered from interviews included views and experiences, explanations and factual information that addressed the evaluation questions.‌

In total, 55 interviews were conducted with 112 participants. This included 31 partner interviews with 63 participants (representing 25 of 45 SSC partners), and 24 enterprise interviews with 49 participants.‌

4. Observations‌

A site visit to SSC's warehouse was conducted for a physical inspection of practices used in the storage and disposal of mobile devices.‌

5. Administrative data analysis‌

Various datasets from the Enterprise Data Repository were analyzed, including Customer Satisfaction Feedback Initiative (CSFI) survey results and data from service requests, business requests, performance metrics and incidents. Other administrative data was obtained from the Telecommunications Program (including data on zero usage mobile devices, progress on the Fixed Line Rationalization initiative, and financial data) and Operations Management Branch (partner consumption forecasts).‌

6. Client survey‌

The evaluation team designed and administered an online survey of partner organizations in Q3 2022/23 to identify the extent to which Mobile Devices and Fixed Lines supported their evolving digital and program needs. It assisted in validating findings from key informant interviews.‌

Invitations were sent to directors and managers of telephony in the 45 SSC partner organizations. Representatives from 40 partners responded to the survey, giving a response rate of 89%.‌

Limitations‌

The limitations for the findings in this evaluation concerned the lack of baseline data, timing issues, gaps in administrative data, insufficient time since the introduction of ESM to assess its longer-terms impacts, and the historical nature of data collection in a period of rapid change.‌

1. There was a lack of baseline information for key program outcomes‌

This was the first evaluation that had been conducted for the Telecommunications Program. Consequently, there was a lack of baseline information and no target levels of improvement for key program outcomes. This made it difficult to assess performance and progress toward program outcomes.‌

2. The timing of the evaluation coincided with the beginning of ESM‌

The evaluation was conducted during a time of substantial change for the Telecommunications Program. The ways that Mobile Devices and Fixed Lines were delivered, funded and regulated were in the process of being transformed. Specifically, ESM was launched in the middle of data collection. There was some trial and error with the introduction of new telephony processes, roles and responsibilities in the first few months of the rollout of ESM. Additionally, proposed changes to telephony standards were under development during the evaluation period. As a result, it was difficult to determine whether some of the impacts on the services and criticisms of SSC performance raised by partners reflected merely "growing pains" associated with the changes or whether they represented systemic issues.‌

3. There were administrative data limitations‌

There were gaps in the administrative data. For example, some Centrex line counts were missing and PBX line counts were absent until September 2021 in the dataset that documented phone line counts by department. Categories for mobile device line counts changed over time and mobile device data was missing entirely from the EDR for the 2019 calendar year. ‌

Furthermore, the evaluation team observed differences between line count data contained in the EDR and line count data used in reports. The evaluation team attempted to compensate for these discrepancies by adjusting for suspended mobile devices. Once suspended mobile devices were excluded from EDR data, the residual difference between line counts in the EDR versus program reports fell to between 1.3 and 0.6%. Suspended mobile devices did not have access to the cellular network and could not make phone calls, send text messages or use data. However, these devices still had phone numbers associated with them, and SSC still had to pay vendor fees for suspended devices. Because SSC had to pay for suspended devices, the evaluation team felt it was important to include them in line count data when possible. ‌

To clarify, this report explicitly explains when suspended mobile devices are included in line count data and when they are excluded. Despite these limitations, the evaluation was able to make general observations regarding trends over time.‌

4. It was too soon for a full assessment of ESM impacts ‌

After the completion of data collection for this evaluation, additional requirements were introduced. Specifically, there was significant interest in the extent to which the Telecommunications Program had achieved enterprise-level objectives and how well the program was performing under the 4 ESM pillars (standards, governance, planning and funding). Additional documents and administrative data were analyzed, and existing interview notes were re-analyzed through this lens. The evaluation also conducted additional interviews with SSC staff. This approach generated additional findings for the report. ‌

Ultimately, since ESM was implemented in April 2022, it was too soon to determine whether ESM achieved its intended impacts on the effectiveness and efficiency of enterprise Mobile Devices and Fixed Lines. ‌

5. The evaluation represents a snapshot in time of the services between January and November 2022‌

Evaluation findings represent the status of Mobile Devices and Fixed Lines at the time of data collection. Most data collection took place between January and November 2022, during a time of substantial change for the services. During this time, SSC responded quite rapidly when they detected negative impacts on the services. This rapid response was a great reflection on SSC's commitment to providing excellent client-centric service. However, because of this, what was true and reported during the time of data collection may no longer reflect the current situation as of July 2023. ‌

There were 2 known examples where important findings may be outdated. They are summarized below:‌

1. There were software errors and other issues with Onyx that decreased efficiency for partners and SSC.

Onyx, a new ITSM tool, was implemented in June 2021 to replace legacy ITSM tools. It was intended to improve how SSC served its partners by putting in place a single self-service portal to support service requests and incident reporting. Internal interviewees expected that Onyx, when working optimally, would increase data quality and transparency related to partner requests.‌

During key informant interviews in May and June 2022, partners consistently reported that Onyx did not improve their service request process. In fact, many partners still perceived Onyx to be a barrier to efficiency in fall 2022. ‌

Partners reported that known errors in the software had not been actioned by SSC in a timely manner. Some of these errors, such as the inability to input correct postal codes, were directly related to shipping errors for mobile phones. Even though the postal code error was identified in October 2021, it was not addressed until June 2022. Some partners reported there were still minor postal code issues in fall 2022. ‌

Partners also reported that Onyx users experienced issues accessing information on the status of their tickets. Some users had their tickets cancelled due to missing information, but were not told what information was missing.‌

Furthermore, at the original time of data collection, some partners were not yet onboarded to Onyx and the use of old ITSM tools was a barrier to efficiency for SSC. When a partner submitted a request on their old ITSM tool, an SSC employee had to copy the information into Onyx before it could be actioned. This created a need to train employees in additional ITSM tools, created bottlenecks and inserted more steps in the process.‌

Without another round of external consultations, it is not clear whether all of these issues have been addressed from the perspective of SSC partners.‌

2. When ESM was initially implemented, SSC's response time to partner service requests suffered.

Before the implementation of ESM, SSC generally fulfilled mobile phone service requests within the service delivery standard of 5 business days. However, when ESM was implemented, service delivery was not timely and service standards for requests were not met by SSC. Service delivery delays reached their peak in June 2022, with the average mobile device request being fulfilled in 26 business days. Service request fulfillment time remained high up to mid-December, averaging between 18 and 23 business days to fulfill.‌

There were a range of impacts from these delays. One partner reported they were unable to provide some employees working in rural communities with mobile phones. These employees were forced to use their own devices for phone calls and were unable to use multi-factor authentication for Microsoft Teams or other M365 programs. This impeded their ability to access tools required for work, causing delays in providing service to the public and creating difficulties in reaching their managers.‌

By March 2023, the mobile device provisioning team had decreased request fulfillment time to about 6 business days, which was very close to the standard.‌

Appendix C: Notional logic model for the evaluation of Mobile Devices and Fixed Lines

Long description

The notional logic model for the evaluation of mobile devices and fixed lines illustrates the activities and outputs of the services and their outcomes. The logic model uses arrows to show how activities and outputs are expected to contribute to intended outcomes, and how immediate and intermediate outcomes are expected to contribute to long term and ultimate outcomes. The diagram also distinguishes between enterprise outcomes and service outcomes. Assumptions and external factors that impact the Telecommunications Program and its intended results are listed.‌

Activity 1: Engage and collaborate with the Office of the Chief Information Officers of Canada and other governing bodies to promote better stewardship of resources across the Government of Canada. This activity contributes to output 1 (IT policies and standards).‌

Activity 2: Design client-centric services, including:‌

• consulting with partners, industry, and accessibility experts
• planning, sourcing, designing, and procuring services

This activity contributes to output 2 (strategies, frameworks, and roadmaps) and output 3 (detailed communication to partners).‌

Activity 3: Consolidate, rationalize, standardize, and modernize telecommunication services, devices, and infrastructure, including:‌

• eliminating unused phone lines
• migrating legacy services to modern enterprise services
• replacing end-of-support devices

This activity contributes to output 3 (detailed communications to partners) and output 4 (service update and forecast reports).‌

Activity 4: Monitor telecommunication networks, devices, consumption, and spending and assess for continued need, including:‌

• validating phone lines
• consolidating and analyzing data

This activity contributes to output 4 (service update and forecast reports) and output 5 (modern and accessible services).‌

Activity 5: Maintain and operate telecommunication infrastructure and services, including:‌

• delivering client-centric services
• fulfilling service and business requests
• resolving incidents and problems

This activity contributes to output 5 (modern and accessible services).‌

Output 1: IT policies and standards. This output contributes to immediate outcome 1.1 (partners have increased understanding of GC IT priorities)‌

Output 2: Strategies, frameworks, and roadmaps that describe the evolution of telephony. This output contributes to immediate outcome 1.1 (partners have increased understanding of GC IT priorities).‌

Output 3: Detailed communications to partners, including:‌

• communiques
• frequently asked questions
• presentations
• process descriptions
• roles and responsibilities
• other reports

This output contributes to immediate outcome 1.1 (partners have increased understanding of GC IT priorities).‌

Output 4: Service update and forecast reports, including:‌

• consumption reports
• finance reports
• service request reports
• business request reports
• service reviews

This output contributes to immediate outcome 1.1 (partners have increased understanding of GC IT priorities).‌

Output 5: Modern and accessible mobile device and fixed line services. This output contributes to immediate outcome 2.2 (partner infrastructure has an improved ability to support modern telecommunication solutions) and immediate outcome 2.3 (users benefit from accessibility features).‌

Immediate outcome 1.1: Partners have increased understanding of GC IT priorities and greater buy-in for the enterprise approach. This is an enterprise outcome. This outcome contributes to immediate outcome 1.2.‌

Immediate outcome 1.2: Partner IT and business plans have increased alignment to the enterprise approach. This is an enterprise outcome. This outcome contributes to immediate outcome 2.1.‌

Immediate outcome 2.1: Partners have increased adoption of common enterprise telecommunication infrastructure, services and standards. This is an enterprise outcome. This outcome contributes to immediate outcome 2.2.‌

Immediate outcome 2.2: Partner infrastructure has an improved ability to support modern, enterprise telecommunication solutions. This is a service delivery outcome. This outcome contributes to intermediate outcome 3.2 (GC telecommunication services are increasingly digitalized and automated), intermediate outcome 3.3 (public servants have increased ability to collaborate with colleagues) and intermediate outcome 4.2 (partners experience improved services quality).‌

Immediate outcome 2.3: Users benefit from accessibility features. This is a service delivery outcome. This outcome contributes to intermediate outcome 3.3 (public servants have increased ability to collaborate with colleagues).‌

Intermediate outcome 3.1: GC telecommunication infrastructure has decreased complexity and reduced reliance on independent and legacy telecommunication infrastructure and services. This is an enterprise outcome. This outcome contributes to intermediate outcome 4.1 (GC telecommunication services have increased operational efficiency) and longer-term outcome 5.1 (improved stewardship of telecommunication resources).‌

Intermediate outcome 3.2: GC telecommunication services are increasingly digitalized and automated. This is an enterprise outcome. This outcome contributes to intermediate outcome 4.1 (GC telecommunication services have increased operational efficiency) and intermediate outcome 4.2 (partners experience improved service quality).‌

Intermediate outcome 3.3: Public servants have increased ability to collaborate with colleagues and benefit from increased productivity. This is a service delivery outcome. This outcome contributes to intermediate outcome 4.2 (partners experience improved service quality).‌

Intermediate outcome 4.1: GC telecommunication services have increased operational efficiency. This is an enterprise outcome. This outcome contributes to longer-term outcome 5.1 (improved stewardship of GC telecommunication resources) and longer-term outcome 5.2 (SSC has an improved ability to support and respond to growing telecommunication demand).‌

Intermediate outcome 4.2: Partners experience improved service quality, including:‌

• improved customer experience
• increased timeliness, reliability, and security
• improved end-user experience and satisfaction with partner telecommunication teams

This is a service delivery outcome. This outcome contributes to ultimate outcome 6.2 (the GC has increased IT support to enable the delivery of GC programs and services).‌

Longer-term outcome 5.1: Improved overall stewardship of GC telecommunication resources, including reduced costs and risks. This is an enterprise outcome. This outcome contributes to ultimate outcome 6.1 (the GC has consolidated enterprise IT infrastructure that is available anytime, anywhere, and to anyone who needs them).‌

Longer-term outcome 5.2: SSC has an improved ability to support and respond to growing telecommunication demand. This is a service delivery outcome. This outcome contributes to longer-term outcome 6.2 (the GC has increased IT support to enable the delivery of GC programs and services).‌

Ultimate outcome 6.1: The GC has consolidated enterprise IT infrastructure that is available anytime, anywhere, and to anyone who needs them. This is an enterprise outcome.‌

Ultimate outcome 6.2: The GC has increased IT support to enable the delivery of GC programs and services in the digital age for all Canadians. This is a service delivery outcome. ‌

Assumptions of the logic model are:‌

• partners and SSC collaborate to modernize telecommunication infrastructure and services
• GC policies and guidance are respected and implemented
• vendors are able to respond based on established service standards
• resources are available to support the modernization of telecommunication infrastructure and services
• adequate processes are in place to ensure efficiency, service quality, and reduce risk

External factors are:‌

• interdependencies with other branches (for example, CFOB, CTOB, NSSB, CSB), vendors and partners may impact implementation and risks

Appendix D: Glossary of terms ‌

Centrex (Central Exchange)

A telephone switching system that is provisioned with equipment owned by telephone companies.‌

Enterprise Service Model (ESM)

A new funding and service model for SSC enterprise services that centralizes and aligns funding under the Treasury Board Policy on Service and Digital. Phase 1 was implemented in April 2022. ESM has 4 pillars: standards, governance, planning and funding.‌

Fixed Line Rationalization (FLR) Initiative

An initiative that started in 2020 with a goal to decrease the number of non-essential fixed line phones among SSC partner organizations.‌

Information Technology Service Management (ITSM)

The process by which IT teams manage IT services.‌

Multi-Factor Authentication (MFA)

A security method that verifies the user logging into an application, online account, VPN or other resource. It requires 2 or more verification factors (a password and another method to confirm identity, often using a separate device) to gain access to the resource. It protects data from being accessed by unauthorized individuals.‌

Onyx

A new IT service management (ITSM) tool. SSC started implementing Onyx in fall 2021. SSC's goal is to migrate all partners to Onyx to consolidate and streamline partner requests.‌

Outcome

Changes or consequences that are directly attributable to products, services or initiatives within a program. Outcomes can be immediate, intermediate or long-term, depending on the time horizon needed for the changes to take place.‌

Output

Direct products and services that come from the activities of an organization, program or initiative. Outputs are usually within the control of the organization.‌

Performance indicator

A qualitative or quantitative means of measuring an output or outcome. Performance indicators gauge the performance of an organization or program against its expected results.‌

Performance Information Profile (PIP)

A document that identifies the performance information for each program in the program Inventory‌

Private Branch Exchange (PBX)

A telephone switching system owned by an organization (instead of a telephone company). It allows phone calls between phones internal to the organization. PBX also provides connections to the public switched telephone network (PSTN) to enable phone calls external to the organization. PBX allows 2 or more stations to directly connect without using the PSTN.‌

Program

Individual or groups of services and/or activities that are managed together within a department or agency and focus on a specific set of outputs, outcomes or service levels.‌

Program inventory

Identification of all the programs within a department or agency. It describes how resources are organized to contribute to the organization's core responsibilities and results.‌

Public Switched Telephone Network (PSTN)

Infrastructure and services for public telecommunication. It is the aggregate of the world's circuit-switched telephone networks operated by national, regional or local telephone companies. PSTN consists of phone lines, fiber optic cables, microwave transmission links, cellular networks, communications satellites and undersea telephone cables. This infrastructure is interconnected by switching centres that enable most phones to communicate with each other.‌

Subscriber Identity Module (SIM) card

an integrated circuit used to identify and authenticate subscribers on mobile devices. Most cellular networks require users to have a SIM card for network access.‌

Softphones

Software that allows users to make and receive phone calls over a laptop, tablet or smartphone. Examples of softphones include Microsoft Teams and Skype.‌

Voice over Internet Protocol (VoIP)

Technology that allows users to make voice calls over a broadband Internet connection instead of an analog phone line.‌

Appendix E: Glossary of acronyms

CE

Client Executive‌

Centrex

Central Exchange‌

CIO

Chief Information Officer‌

CSFI

Customer Satisfaction Feedback Initiative‌

EMDM

Enterprise Mobile Device Management‌

ePET

Enterprise Price Estimation Tool‌

ESM

Enterprise Service Model‌

FLR

Fixed Line Rationalization‌

GC

Government of Canada‌

GC EARB

Government of Canada Enterprise Architecture Review Board‌

IM

Information Management‌

IT

Information Technology‌

ITSM

Information Technology Service Management‌

MFA

Multi-Factor Authentication‌

PBX

Private Branch Exchange‌

PIP

Performance Information Profile‌

PSTN

Public Switched Telephone Network‌

SIM card

Subscriber Identity Module card‌

SSC

Shared Services Canada‌

VoIP

Voice over Internet Protocol‌

Wi-Fi

Wireless Fidelity‌