Evaluation of the Canada Pension Plan Service Improvement Strategy

On this page

• Alternate formats
• List of figures
• List of tables
• List of acronyms
• Executive summary
  • Introduction and methodology
  • Main findings
• 1. Introduction
• 2. Canada Pension Plan Service Improvement Strategy profile
  • 2.1 Canada Pension Plan
  • 2.2 Modern service delivery
  • 2.3 Canada Pension Plan Service Improvement Strategy profile overview
• 3. Evaluation approach
  • 3.1 Objectives and scope
  • 3.2 Methodology
• 4. Evaluation findings
  • 4.1 Was the Canada Pension Plan Service Improvement Strategy profile implemented in a cost-effective manner
  • 4.2 What are the lessons learned through client consultations, engagement and client experience testing
  • 4.3 To what extent did the Canada Pension Plan Service Improvement Strategyachieve its expected outcomes
  • 4.4 To what extent did the Canada Pension Plan Service Improvement Strategy contribute to Canada Pension Plan service-delivery outcomes (“Clients receiving high-quality, timely and efficient services that meet their needs”)
• 5. Conclusions
• Appendix A – Canada Pension Plan overview
• Appendix B – List of initiatives and completion dates
• Appendix C – Lines of evidence and limitations 

Alternate formats

Evaluation of the Canada Pension Plan Service Improvement Strategy [PDF- 1.8 MB]

Large print, braille, MP3 (audio), e-text and DAISY formats are available on demand by ordering online or calling 1 800 O-Canada (1-800-622-6232). If you use a teletypewriter (TTY), call 1-800-926-9105.

List of figures

• Figure 1: Percentage of expenditures by CPP benefit type (2020 to 2021)
• Figure 2: CPP SIS objectives, expected benefits and reviewed initiatives (completed before December 2020)
• Figure 3: Project cost variance
• Figure 4: Government of Canada digital standards
• Figure 5: Expected vs. achieved uptake for CPPD online applications for clients eligible to apply online (2019 to 2020) to (2021 to 2022)
• Figure 6: CPPD Toolkit in numbers (January 2019 to March 2020)
• Figure 7: Expected vs. achieved uptake for CPP Retirement for clients eligible to apply online (2017 to 2018) to (2021 to 2022)
• Figure 8: Proportion of CPP and CPPD clients using online channels at the Aware, Apply and Follow-up stages
• Figure 9: Proportion of CPPD Client Experience Survey respondents rating ease of application through digital services (4 or 5 on a 5-point scale)
• Figure 10: Proportion of CPP and CPPD Client Experience Survey respondents rating ease of application (4 or 5 on a 5-point scale)
• Figure 11: Proportion of CPP and CPPD Client Experience Survey respondents rating the effectiveness of the benefit process (4 or 5 on a 5-point scale)
• Figure 12: Proportion of CPP and CPPD Client Experience respondents rating the program service experience (4 or 5 on a 5-point scale)
• Figure 13: Departmental Service Standard results for CPPD applicants (2021 to 2022)
• Figure 14: Satisfaction for contacted CPPD clients by Service Canada prior to a decision vs. non contacted clients (2020 to 2021 to 2019 to 2020)
• Figure 15: Proportion of CPP and CPPD respondents strongly agreeing or agreeing that the amount of time it took, from gathering information to receiving a decision, was reasonable
• Figure 16: Estimated CPP SIS cost efficiencies before project implementation vs. at project closure
• Figure 17: Percentage of persons with disabilities having difficulty accessing Government of Canada services

List of tables

• Table 1: Overall project cost ($)
• Table 2: (2020 to 2021) CPPD client attribute ratings (4 or 5 on a 5-point scale) and percentage point changes from (2019 to 2020)
• Table 3a: CPP SIS initiatives and implementation dates
• Table 3b: CPP SIS initiatives and implementation dates – Objective 2: Excellence in performance and results
• Table 3c: CPP SIS initiatives and implementation dates - Objective 3: Excellence in program stewardship
• Table 4: Scale used to report findings for key informant interviews and focus groups from qualitative studies

List of acronyms

CPP

Canada Pension Plan

CPPD

Canada Pension Plan Disability

CPP SIS

Canada Pension Plan Service Improvement Strategy

e-Services

Online services

ESDC

Employment and Social Development Canada

IT

Information technology

OAS

Old Age Security

Executive summary

Introduction and methodology

This report presents the results of the evaluation of the Canada Pension Plan Service Improvement Strategy (CPP SIS). The CPP SIS implemented a series of information technology (IT) and non-IT service improvement initiatives between 2017 and 2022.

The objectives of this evaluation are to assess the early contribution of CPP SIS project efforts toward outcomes pertaining to the improvement of client service, performance and results and stewardship of resources. The review of outcomes covers initiatives implemented between March 2017 and December 2020^{Footnote 1}.

The evaluation also reports on challenges and key lessons learned by examining the implementation and delivery of the full scope of planned initiatives.

Lines of evidence

• Key informant interviews: 30 semi-structured interviews with 38 Government of Canada stakeholders (management members, National Headquarters officers, benefit officers and medical adjudicators in regional processing centres)
• Administrative data: Analysis of administrative documentation collected during the implementation of the initiatives including financial data
• Document review: Analysis of approximately 100 documents from internal and public web content
• Literature review: Presents different approaches used to increase client service excellence and reduce costs

Main findings

Canada Pension Plan Service Improvement Strategy profile implementation

The CPP SIS was implemented and delivered within expected scope and budget.

Factors that contributed to successful implementation included:

• communication and engagement with the appropriate stakeholders and CPP SIS
• flexibility regarding the management of its individual initiatives

Challenges related to implementation included:

• conflicting departmental priorities for IT projects causing delays
• staff turnover resulting in a loss of expertise
• the absence of a dedicated team responsible for performance measurement throughout the project

Lessons learned through client consultations, engagement, and client-experience testing

• Client consultations need to be included early in the development of initiatives
• Engagement and client-experience testing brought:
  • insights to better support online uptake
  • a more streamlined and client-centric CPP Disability (CPPD) application
  • process improvements to decrease decision times for CPPD adjudications

Achievement of expected outcomes

Objective 1: Excellence in client service

There is some early evidence indicating that reviewed initiatives expanded the CPP online channel, such as:

• improved CPP service experience by increasing take-up rates for online applications and online services
• over time, an increasing share of CPP and CPPD Client Experience Survey respondents agreed that the ability to complete steps online made the process easier

Objective 2: Excellence in performance and results

There is some evidence suggesting that initiatives implemented contributed to improved performance and results through increased speed of service and reduced waiting times and adherence to service standards. For instance:

• process-improvement initiatives did appear to contribute to improved speed of service for CPPD application adjudication
• service standards for pensions and benefits were almost all exceeded by 2021 to 2022

Objective 3: Excellence in program stewardship

The evidence shows that initiatives implemented contributed to the integrity, risk-management and sustainability of the CPP program to some extent. Notably, the CPP SIS expected estimated cost efficiencies were exceeded by March 2022.

However, there was a lack of data to determine the CPP SIS impact of decreased manual work, workload-management improvements and actual cost efficiencies.

Looking ahead and building on activities implemented under the CPP-SIS, the following should be considered to continue strengthening the delivery of the CPP program:

• CPP Disability (CPPD) clients were still experiencing difficulties in assessing services and applying for benefits
• continued barriers to accessibility (digital and non-digital)

From these main findings, the following observations can be drawn:^{Footnote 2}

• innovative methods and approaches, such as user-centred design, could contribute to identify and reduce barriers for CPPD clients and vulnerable populations
• improved performance metrics on cost savings, productivity tracking and reporting could support to improve project decision-making

1. Introduction

This report presents the results of the evaluation of the Canada Pension Plan Service Improvement Strategy (CPP SIS).

The CPP SIS implemented a number of information technology (IT) and non-IT service improvement initiatives between 2017 and 2022. Those initiatives aimed at addressing the overall delivery of the CPP program by:

• delivering simplified, electronic services to Canadians
• increasing the processing automation of CPP benefits
• modernizing internal tools
• maintaining quality of decisions
• increasing efficiency and decreasing operational costs

The objectives of this evaluation are to assess the early contribution of project efforts toward CPP SIS outcomes pertaining to the improvement of client service, performance and results, and stewardship of resources. The review of outcomes covers initiatives implemented between March 2017 and December 2020.

The evaluation also reports on challenges and lessons learned by examining the implementation and delivery of the full scope of planned initiatives.

2. Canada Pension Plan Service Improvement Strategy profile

2.1 Canada Pension Plan

The CPP was established in 1966, as part of Canada’s retirement income system. It is a contributory social insurance program that operates throughout Canada, except in Quebec, where the Quebec Pension Plan provides similar benefits. It provides eligible contributors and their families with income replacement in the event of retirement, disability or death. (Refer to Appendix A for a brief description of the benefits provided under this plan).

The CPP program is delivered through Service Canada, which operates a multi-channel network of more than 600 offices and outreach locations (in-person points of service), 1 800 O-Canada call centres, and the Internet.^{Footnote 3}

In the fiscal year ending March 31, 2021, Employment and Social Development Canada (ESDC) CPP service delivery expenses^{Footnote 4} amounted to $244.4 million.^{Footnote 5} Nearly 40% of those expenses ($94.9 million) were attributed to the delivery of CPP Disability (CPPD) benefits, programs and services.^{Footnote 6}

According to CPP annual statistical data, $51.3 billion of CPP benefits were paid to 6.6 million beneficiaries in 2020 to 2021.^{Footnote 7} This represents an increase of 1.8 million beneficiaries (+27%) since 2011 to 2012.^{Footnote 8} Figure 1 illustrates the distribution of expenditures according to CPP benefit type.

• Source: ESDC. Administrative data.
• Note: Percentages may not add up owing to rounding.

Figure 1 - Text version 

• Retirement (less overpayments): 78,5% of total expenditures
• Disability: 9.4% of total expenditures, including:
  • Disability (including Post-Retirement Disability Benefits): 93% of disability benefits
  • Dependant Children of Contributors with Disabilities: 6.6% of disability benefits
• Survivor: 10.6% of total expenditures, including:
  • Survivor’s Pension: 88.2% of survivor benefits
  • Death Benefit: 7.9% of survivor benefits
  • Dependent Children of Deceased Contributors: 3.9% of survivor benefits
• Post-Retirement Benefits: 1.6% of total expenditures

2.2 Modern service delivery

The Government of Canada continues to work towards improving how it delivers services to the public.^{Footnote 9} Service delivery in the private sector has evolved rapidly through ever-advancing technologies. As a result, clients increasingly expect the delivery of government services to keep pace. This means making services digital, easy to access, available at any time and accompanied by timely assistance.^{Footnote 10}

Modernizing services for Canadians has become a Government of Canada priority over the past 2 decades.^{Footnote 11} The approach to public-sector service delivery has shifted to adopt many of the same private-sector principles, such as digital innovation, automation, streamlining and user-centricity.

In this context, ESDC, through Service Canada, has committed to transforming its service-delivery approach to enhance the service experience.

These efforts are ongoing and range from broad initiatives spanning across multiple programs, such as Benefits Delivery Modernization^{Footnote 12}, to more program-specific strategies, such as the CPP SIS.

2.3 Canada Pension Plan Service Improvement Strategy profile overview

The CPP SIS consists of a series of IT and non-IT initiatives aimed at addressing the overall delivery of the CPP program. It contributed, among others, to enhancing electronic services by streamlining and simplifying processes for CPP and CPP Disability (CPPD) clients.

This directly supports ESDC’s commitment to modernizing delivery of the CPP program by improving services to Canadians, including vulnerable individuals. Also, by maximizing the use of technology with emphasis on electronic services and automation to simplify processes and generating efficiencies (cost avoidance) in the processing of CPP benefits.

The CPP SIS was implemented to respond to the following concerns:

• significant pressure on the Department’s ability to process CPP payments in a timely and cost-effective manner, due to aging population
• evolving client-service expectations leading to a demand from Canadians for their government service experience to be online and on par with that of the private sector
• audit findings from the fall 2015 Auditor General Report^{Footnote 13} on the CPPD program. The audit recommended significant changes to improve processes and timeliness for applications, reconsiderations and appeals

The CPP SIS was launched in June 2016, with an intended timeline of 3.5 years. Owing to competing priorities, the CPP SIS was extended by 2 years and the last initiative was launched in June 2022.^{Footnote 14} The total project cost was estimated to be $78.2 million in 2016.

The cost was revised to $75.8 million in 2017, and again to a lower cost of $75.2 million in 2019.^{Footnote 15} Appendix B provides a description of the initiatives and completion dates. You can also refer to Figure 2 for an overview of expected outcomes and reviewed initiatives.

Figure 2 - Text version

Objective 1: Excellence in client service

An overall improvement of service experience to Canadians as a result of delivering user-friendly electronic services through the online channel.

Benefit A: Improved CPP service experience.

Benefit B: Reduction in client effort to access CPP benefits and provision.

In-scope initiatives:

• 1.1 Expand and Promote e-Services
  • 1.1.3 Child Rearing Provision
  • 1.1.4 Voluntary Tax Withhold
  • 1.1.9 View Status of CPP and OAS Application
• 1.2 Online CPPD Application
• 1.4.1 CPP Children’s Benefit (over 18-School Attendance)
• 1.5 Canada Revenue Agency Collaboration
• 1.6 Enable Document Upload for Online CPPD application

Objective 2: Excellence in performance and results

Improving performance and results, especially for high priority work, and ensuring accountability for these results through strengthened service standards.

Benefit C: Increased speed of service and reduced waiting times.

In-scope initiatives:

• 2.1.2 Notice of Entitlement Process Improvement
• 2.2.1 Client Engagement Pilot
• 2.2.2 Terminal Illness and Grave Conditions Pilot
• 2.2.3 Long-Term Disability Insurers Prototype
• 2.3 Implement Workload Management

Objective 3: Excellence in program stewardship

Contributing to long-term CPP program sustainability through reductions in operating costs and/or cost avoidance, and enhanced integrity and risk management capabilities.

Benefit D: Long term and/or program-level integrity improvement and risk management.

Benefit E: Reduction in future operating costs.

In-scope initiatives:

• 3.1 Introduce a personal Access Code Expiry Date to My Service Canada Account (MSCA)
• 3.2 Enhance system Access Control and Audit Trail
• 3.3 Explore & Develop Predictive Analytics
• 3.4.1 CPPD Abbreviated Adjudication Decision Template (Part 1)
• 3.4.2 Pensions Knowledge Management Tool

3. Evaluation approach

3.1 Objectives and scope

The evaluation assesses the early contribution of project efforts to CPP SIS outcomes with respect to improving client service, performance/results and stewardship of resources. It examines how initiatives improved the delivery of CPP benefits, either through streamlining and automating the application process or increasing e services.

The evaluation covers initiatives completed over 2 phases between March 2017 and December 2020:^{Footnote 16}

• phase 1 (fiscal year from 2016 to 2017 and 2017 to 2018 ) mainly focused on early online services and improvements related to simplifying and streamlining CPPD application processes
• phase 2 (initiatives implemented by December 2020) included processing system enhancements and initiatives to improve internal decision-making tools for benefit applications

3.2 Methodology

The evaluation addresses 4 main questions:

1. was the CPP SIS implemented and delivered in a cost-effective manner?
2. to what extent did the CPP SIS achieve its expected outcomes?
   • Excellence in client service
   • Excellence in performance and results
   • Excellence in program stewardship
3. what are the lessons learned in terms of client consultations, engagement, and client experience testing as part of the prototype, design and implementation of CPP SIS initiatives? 
4. in relation to ESDC’s Departmental Plan, to what extent did the CPP SIS contribute to CPP service-delivery outcomes (clients receiving high-quality, timely and efficient services that meet their needs)?

Lines of evidence

The evaluation uses a multiple lines of evidence approach. Appendix C presents details of the evaluation methodology and the limitations.

• Document review
• Administrative data analysis
• Literature review
• Key informant interview

4. Evaluation findings

4.1 Was the Canada Pension Plan Service Improvement Strategy profile implemented in a cost-effective manner

The review of internal documents and view of key informants^{Footnote 17}. indicate that the CPP SIS project was implemented and delivered within scope and budget.

To determine if the program was implemented in a cost-effective manner, elements related to the design and delivery of the program were examined.

Most of the key informants agreed that the design, delivery and implementation of the CPP SIS was a success. They highlighted that the 18 initiatives included in the scope of this evaluation (as of December 2020) were implemented as initially planned. (See Figure 2 for the list of reviewed initiatives and Appendix B for the summary details)

Many key informants commented that the online services were successfully implemented. A few of them highlighted online services as the fundamental deliverable for the CPP SIS. Overall, CPP SIS added to the success of the CPP online Retirement pension application. The project expanded and promoted the online application functionalities in an effort to improve take-up. 

I think it was a great success from the design to delivery to implementation. With any new e-Service, there  are always challenges, but I know that just from the usage and client impact feedback, that’s been very positive.

Cost^{Footnote 18}.

Financial information indicated a variance of 12% between actual and baseline Project total costs. As shown in Table 1, the total Project cost of $67,047,213 is about $5 million lower compared to the planned budget of $75,250,344 million.^{Footnote 19}.

Figure 3 - Text version

• Planned budget: $75 million
• Project cost: $67 million
• Project cost equal to 88% of Planned budget

Table 1: Overall project cost ($)

Cost Categories	Cost Baseline	Actual Cost	Estimate to Complete	Total Cost	% Variance
Contingency	2,546,601	0	489,193	489,193	- 80.79%
Total direct costs	54,729,980	47,565,267	554,663	48,119,930	-12%
Total indirect costs	20,520,364	18,494,939	432,404	18,927,343	- 8%
Full cost	75,250,344	66,060,206	987,067	67,047,273	- 12%

• Source: Administrative data provided by ESDC Chief Financial Officer Branch.
• Note: (1) Costs were extracted from SAP financial management software on August 18, 2022.
• (2) Estimate to Complete” represents outstanding transactions to be processed.
• (3) Indirect costs include estimated corporate costs, employee benefit plans and taxes.

Total project costs (Phases 1 and 2) were revised twice throughout the implementation. The first revision prior to Phase 2 implementation highlighted that the total project cost declined from $78,234,966 to $75,809,43. The second revision included in the Phase 2 extension request was $75,250,344.^{Footnote 20}.  This amount corresponds to the cost baseline in Table 1.

Variance in project costs resulted from a variety of factors, including:

• early implementation of Phase 1 initiatives
• project efficiencies through joint CPP/Old Age Security (OAS) online service development and implementation^{Footnote 21}
• effective financial management ensuring that expenditures and work performed could be adjusted during periods of delays^{Footnote 22}

Time

The initial timing for CPP SIS implementation was divided into 2 phases. Phase 1 operated from March 2016 to December 2017 and Phase 2 operated for the remainder of 2017 to 2018 until March 2020.^{Footnote 23}

An extension was requested for Phase 2 from April 2020 to June 2022. The extension included a request to re-profile unspent 2019 funds to deliver the outstanding initiatives.^{Footnote 24} There was no overall change to project costs or scope associated with the Phase 2 extension.

The document review found evidence to support that Phase 1 implementation was on time.^{Footnote 25} According to project documents, Phase 2 IT-dependent initiatives were not all completed on time owing to competing priorities and resource gaps.^{Footnote 26}

Most key informants confirmed that the CPP SIS and its individual initiatives were implemented on time and according to schedule. A few key informants noted that the extension from March 2020 to June 2022 (namely the Phase 2 extension) was the result of shifting IT resources as the Department prioritized the OAS modernization strategy over the CPP SIS. In short, the extension was not owing to delays created by the CPP SIS.

Timeline by Canada Pension Plan Service Improvement Strategy profile’ objectives and project phase

As shown in Figure 2, each objective sought by the CPP SIS was supported by various initiatives. Among the initiatives reviewed as part of this evaluation, the document review found that:

• initiatives supporting Objective 1- Excellence in Client Service were implemented either on time or ahead of schedule. The online CPPD application was the only initiative being slightly delayed by a couple of months to accommodate additional testing^{Footnote 27}
• the majority of initiatives supporting Objective 2- Excellence in Performance and Results and 3- Excellence in Program Stewardship were implemented on time. The document review found evidence indicating that the Pensions Knowledge Management Tool^{Footnote 28} was delayed due to competing priorities and delays in resource allocation, ending in March 2020 instead of September 2019.^{Footnote 29} All other initiatives were implemented on time

Scope

The document review did not reveal any evidence indicating that the initiatives included in this review changed in scope. Moreover, most key informants agreed that the CPP SIS project was generally delivered as planned, despite a few key informants reiterating the need to shift timing and priorities at certain times throughout the project.

As per the Project Closure Report (August 2022), a total of 31 initiatives were implemented under the CPP SIS out of a total of 36 initiatives. Two new initiatives related to online services functionality were added to the CPP SIS Project, and 5 were removed from the original proposal.^{Footnote 30} (Summary details on initiatives are presented in Appendix B).

New initiatives

• 1.1.6 Credit Split Application
• 1.6 Enable Document Upload for Online CPPD Application

Initiatives removed (de-scoped)

• 1.1.5 Foreign Benefits
• 2.1.1 Automate Record of Earnings
• 2.1.3 Enhance Pension Sharing Process
• 2.1.4 Improve Payment Exception Processing
• 3.4.3 Automate Enhanced CPPD Adjudication Decision Template

Key informants indicated that the 5 initiatives that were removed were deemed no longer necessary/relevant by operational experts in the evolving context. These initiatives were out of the evaluation scope (March 2017 to December 2020).

I think for the most part the delivery of the initiative went according to plan. As far as the timing of its delivery, this is a huge department with several large projects running concurrently, so there was a need to be mindful that some flexibility would be required. (…) At a project’s outset, we try to anticipate as many of these things as possible and build a project that’s nimble enough to be able to navigate around any constraints as they emerge. I think that’s one of the things we did well.

Factors of success and challenges

Many key informants agreed that the CPP SIS and its individual initiatives were delivered on time, within budget and within its planned scope.

Key factors that contributed to this outcome were ongoing communication and engagement with appropriate internal stakeholders, including Service Canada regional offices throughout the country. Main elements highlighted by key informants supporting the engagement and communication included:

• experienced business teams
• working groups across business lines
• IT and operational support
• regional collaboration
• fully cross-functional teams

On this matter, the review of internal documents shows that the project had been constituted as self-contained initiatives that could be prioritized and shifted as needed. This included an iterative costing approach supported by all directorate areas involved in the implementation, breaking the project down into smaller parts for estimation.

According to the project’s document review^{Footnote 31}, this approach ensured that each area could provide for the precise level of resources required and salary costs associated with the completion of their assignments.

It also supported timeline management for individual initiatives and the overall project schedule, and was particularly useful for tracking budgets when using agile project management methodology.^{Footnote 32}

Some of the challenges raised by key informants included:

• competing priorities within the Department (for example, the OAS Service Improvement Strategy)
• working across business lines and branches with different priorities
• staff turnover resulting in a loss of expertise

The best practices and lessons learned that were leveraged throughout the course of the project management and implementation were collected through the document review. The CPP SIS Lessons Learned Report (2016 to 2022) and the key informant interviews were the primary sources of information.

Human resources

Lessons learned

• The main challenges identified during project implementation were:
  • staff turnover resulting in a loss of expertise
  • resource capacity (especially IT)
  • the lack of a dedicated benefit-realization team for the project’s duration

Best practices

• Perform strategic re-prioritization exercises when faced with changes in project environment
• Seek third-party support to leverage expertise and fill resource gaps^{Footnote 33}
• Ensure dedicated project resources with project-management expertise, including a dedicated team to track project metrics and measure results for the project’s duration

Training and knowledge transfer

Lessons learned

• The CPP SIS included a network-wide effort to identify and mitigate the impact of implementation on operations before national onboarding, such as:
  • training and coaching plan
  • operational-readiness assessment
  • change-management strategy

Best practices

• Rely on comprehensive change-management monitoring based on the complexity of each initiative and anticipate impacts of changes on the organization and staff 
• Ensure timely project handovers to the Operations sector in order to better support the transfer of knowledge and responsibility
• Provide partners/employees with enough time and support to review and master materials and information before implementing the changes

Governance and decision-making

Lessons learned

• Despite general agreement that the multi-layered governance structure was adequate, challenges raised included:
  • competing priorities within the Department and improvements needed to the governance structure for project prioritization and decision-making
  • difficulties encountered when working across branches (for example, Operations, National Headquarters, regions and policy) and coming to agreements

Best practices

• Establish close cooperation (for example, meetings, working groups) at the management level, namely directors and lower levels, to provide additional discussion opportunities n order to help streamline and build consensus
• Develop a clear governance structure in terms of roles and responsibilities to ensure timely and proper engagement from all parties

4.2 Lessons learned through client consultations, engagement and client experience testing

Expanding and promoting e-services

A Client Engagement activity led by the ESDC Innovation Lab was initiated with recent CPP online applicants in March 2016.

The purpose of this activity was to identify opportunities to increase online CPP applications through client and staff consultations. The following insights were gained from the activity:

• the website needs to be intuitive and easy to use, including plain language, variable text and icons/buttons that direct clients to the information they need
• registering for the My Service Canada Account is perceived to be confusing and difficult. Clients do not necessarily have all the information they need to make an informed decision to apply to CPP
• computer skills are not a predictor of whether application are completed online. It is important to ensure that online applications are easy to find
• clients are frustrated by having some CPP applications online and others only available as hard copy
• an advisory body should be established, with face-to-face meetings to support CPPD renewal^{Footnote 34}

Canada Pension Plan Disability program renewal

Some of the CPP SIS initiatives contributed to the renewal of the CPPD program to support improved decision-making, timeliness of applications and adjudication, and program openness with clients.^{Footnote 35}

In 2016, the Project Execution team facilitated a CPPD client and stakeholder engagement event in Gatineau, Quebec. The goal was to gather feedback on planned CPPD program improvements and seek advice on establishing a sustainable, ongoing mechanism to gather feedback.^{Footnote 36} The event led to a number of key findings:

• clients find the application complex and difficult to complete
• awareness of the CPPD program is limited and outreach efforts should be considered
• status updates on a client’s file are too general and require more accuracy
• clients face unique challenges to accessing the program
• expectations and information about the CPPD program are not well communicated (including wait times, support tools, tax implications and reconsideration deadlines)
• clients perceive client calls to be disadvantageous to their application success

Client and stakeholder roundtable

The Client and Stakeholder Roundtable was renewed and feedback was sought on specific CPPD initiatives in 2017, including revised application, medical, and Request for Reconsideration forms. The following list highlights the feedback gained from the Client and Stakeholder Roundtable regarding the revised CPPD application package:

• roundtable participants agreed that the application and reference guide were functional and practical
• roundtable participants expressed concern regarding the tone and clarity of language (not client-centric and overly bureaucratic)
• clients and stakeholders suggested:
  • adding a comment box to yes/no questions so that applicants could clarify their responses
  • including additional information on program eligibility
  • explaining the Minimum Qualifying Period
  • providing a separate Terminal Illness Form so as not to overwhelm clients^{Footnote 37 Footnote 38 Footnote 39}

The Client and Stakeholder Roundtable was instrumental in helping ESDC understand program challenges from the client’s perspective and guiding the development and revision of online CPPD applications.^{Footnote 40} However, some key informants explained that not all recommendations could be implemented.

The consultations were excellent and valuable in gathering insights and informing the design of the streamlined [CPPD] application. However, it was not possible to implement all of the client suggestions because we needed to align with legislation and the policy framework.

Long-Term Disability Insurers Prototype

Initiative 2.2.3 Long-Term Disability Insurers Prototype (Prototype) involved collecting feedback from Long-Term Disability insurers. The following insights were gained through the survey:

• there is no standard among Long-Term Disability insurers when it comes to advising clients to apply for CPPD
• Long-Term Disability insurers offered a range of assistance to clients applying for CPPD, from fully supporting clients through the process to supporting when requested by clients
• the Prototype required insurers to adjust their internal processes, which some organizations perceived to be more time consuming
• having a Service Canada contact was beneficial and a practice that some would like to see continued beyond the pilot^{Footnote 41}

Putting people first: user-centred service delivery

The literature review suggested that a central component of a service-oriented government with a user-centred approach is understanding the needs of users and building services, programs and policies around users rather than concerns about organizations or silos.^{Footnote 42}

In terms of good practices and guidance, the Accessible Canada Act was enacted in 2019, along with a government-wide strategy entitled Nothing Without Us.^{Footnote 43} The Act requires government departments to increase the inclusion and participation of persons with disabilities or functional limitations when designing and delivering programs and services.

In addition, the Government of Canada Digital Standards Playbook contains evergreen standards for designing and delivering public services.^{Footnote 44} The playbook was co-created with stakeholder groups and will continue to evolve in the Canadian context.

Figure 4 illustrates the Playbook digital standards supporting ESDC’s commitment to better understanding the reality of how people live with a disability. It also helps to identify the service barriers that pose challenges when trying to access government programs and services.^{Footnote 45}

Evidence suggests that the CPP SIS has considered most of those principles with extensive consultations and design with users.

• Source: Canada. Digital Standards (Poster).

Figure 4 - Text version

• Design with users
  • Research with users to understand their needs and the problems we want to solve. Conduct ongoing testing with users to guide design and development
• Iterate and improve frequently
  • Develop services using agile, iterative and user-centred methods. Continuously improve in response to user needs. Try new things, start small and scale up
• Build in accessibility from the start
  • Services should meet or exceed accessibility standards. Users with distinct needs should be engaged from the outset to ensure what is delivered will work for everyone
• Work in the open by default
  • Share evidence, research and decision making openly. Make all non-sensitive data, information, and new code developed in delivery of services open to the outside world for sharing and reuse under an open licence
• Empower staff to deliver better services
  • Make sure that staff have access to the tools, training and technologies they need. Empower the team to make decisions throughout the design, build and operation of the service
• Collaborate widely
  • Create multidisciplinary teams with the range of skills needed to deliver a common goal. Share and collaborate in the open. Identify and create partnerships which help deliver value to users
• Be good date stewards
  • Collect data from users only once and reuse wherever possible. Ensure that data is collected and held in a secure way so that it can easily be reused by others to provide services
• Use open standards and solutions
  • Leverage open standards and embrace leading practices, including the use of open-source software where appropriate. Design for services and platforms that are seamless for Canadians to use no matter what device or channel they are using
• Design ethical services
  • Make sure that everyone receives fair treatment. Comply with ethical guidelines in the design and use of systems which automate decision making (such as the use of artificial intelligence)
• Address security and privacy risks
  • Take a balanced approach to managing risk by implementing appropriate privacy and security measures. Make security measures frictionless so that they do not place a burden on users

Indeed, a few key informants noted that the client consultations component of the CPP SIS initiatives were a model on how to include clients early in the development of initiatives.

Many interviewees also confirmed that enough time should be set aside for stakeholder engagement (external and internal) during all stages of the initiatives.

Other lessons learned for user-centred service delivery gathered during interviews with informants included the following:

• the client engagement approach for CPPD clients of understanding pain points early on and having clients co design the initiatives was a model for all future initiatives of this kind. Client consultations included not just getting their opinions, but also informing them of application features
• the need to listen to an array of clients with an open mind and understand that it is a time-consuming process. Clients are not homogenous, especially when it comes to CPPD clients living with different types of disabilities. There is a need to hear all of those voices and consider them in the changes that will be made
• make the interaction with clients personable to each client in order to keep them engaged and interested in the process

4.3 To what extent did the Canada Pension Plan Service Improvement Strategy achieve its expected outcomes

Objective 1: Excellence in client service

Has the project improved client service by expanding CPP program service through online channel and providing more user-friendly electronic services and streamlining and simplifying the CPP and CPP Disability?

Benefit A: Improved CPP service experience through the online channel

How the Benefit is planned to be achieved

Creation of an online access point for all CPP applications.

Benefit B: Reduction in client effort to access CPP benefits and provisions

How the Benefit is planned to be achieved

Streamlined client applications.

The evaluation found some early evidence indicating that the implemented initiatives expanded the CPP online channel, but found limited evidence indicating the online CPPD application improved the overall clients’ service experience.

The reviewed initiatives included in Objective 1: Excellence in Client Service were designed to improve the CPP experience by expanding the online channel, providing more user-friendly electronic services, and streamlining CPP and CPPD applications.^{Footnote 46}.

It is important to note that 11 additional initiatives were outside the scope of the evaluation. This review only represents a partial assessment of the full functionality of the online channel and its components. The full list of CPP SIS initiatives with their completion dates is presented in Appendix B.

Canada Pension Plan Disability online application

The online CPPD application was implemented in January 2020, and contributed to an improved client experience through the online channel.

In 2022, CPPD applicants could upload supporting documents for their applications and submit a Request for Reconsideration of the decision. However, they could not read the Department’s decision or be issued documentation online.^{Footnote 47}

Canada Pension Plan Disability online take-up rate

The implementation of the online application appears to be well received by CPPD clients. Although only 5,550 (6%) of all CPPD applications in 2020 to 2021 were submitted via the online channel, the proportion went up to 12% in 2021 to 2022 (Figure 5).^{Footnote 48} This represents 50% more than the expected volume of applications (9,474 actual applications vs. 5,000 expected applications).^{Footnote 49}

• Data Source: ESDC. Administrative data.
• Note: The expected CPPD online application for 2019 to 2020 was based on the initial launch of November 2019.

Figure 5 - Text version

Fiscal year	Expected uptake	Achieved uptake
2019 to 2020	3,000	50
2020 to 2021	4,000	5,550
2021 to 2022	5,000	9,474

The online CPPD application fostered a streamlined process by responding to client inputs and launching the appropriate sections or applications according to the data clients entered.^{Footnote 50}

• A streamlined and simplified paper application, which most CPPD clients use to apply for benefits, was implemented in September 2017
• It was followed by a revised paper application kit in September 2018, a Terminal Illness Application Kit in August 2018 and a Reconsideration Form in August 2018^{Footnote 51}
• A CPPD online toolkit was released in January 2019. It was designed to help potential clients, health care professionals and non-government organizations to access all required program information in a single document^{Footnote 52}
• As of March 2020, the printable CPPD Toolkit had been downloaded 12,987 times and the interactive version had been downloaded 11,389 times, with over 50,000 website visits^{Footnote 53} (Figure 6)

• Source: ESDC. Administrative data.

Figure 6 – Text version

CPPD Toolkit in numbers (January 2019 to March 2020)

• 11,389 downloads of the interactive version
• 12,987 downloads of the printable version
• Over 50,000 website visits

While uptake has exceeded expected results, several key informants from the service-delivery sector noted that uptake of online CPPD applications is still low.

Those key informants mainly attributed the lower number of applications submitted online (12% of total applications in 2021 to 2022) to the applicants not using and/or not being familiar with computers.

Canada Pension Plan online services

The CPP Retirement application was made available online in 2015 prior to the launch of the CPP SIS. In 2017 to 2018, improved online service functionalities were added to the existing CPP Retirement online application in order to increase take up. The availability of new services was promoted through targeted letters, website messaging and messaging provided by telephone and by in-person employees.^{Footnote 54}

Canada Pension Plan Retirement online take-up rate

As reported on the GC InfoBase, the CPP Retirement online application uptake rate has been relatively stable since 2018 to 2019, with a 5% increase in 2021 to 2022.^{Footnote 55} As seen in Figure 7, take up is below expectations and the gap is widening each fiscal year.

Figure 7 – Text version

Fiscal year	Expected uptake	Achieved uptake
2017 to 2018	120,000	98,055
2018 to 2019	160,000	135,153
2019 to 2020	200,000	148,605
2020 to 2021	240,000	150,583
2021 to 2022	280,000	157,780

Overall online Canada Pension Plan interactions

Evidence supporting increased uptake of online CPP services is also available from the Client Experience Survey. The 2020 to 2021 data indicates positive trends among CPP and CPPD respondents. Both respondent groups reported an increased use of the online channel at the Aware, Apply and Follow-up stages, as seen in Figure 8.

In 2020 to 2021, service options for CPP clients were: In-person, online self-service, assisted online service, touchless person-to-person service and mail-in applications.

• Source: Client Experience Surveys (2017 to 2018) to (2020 to 2021).
• Note: The " Apply" stage refers to clients that used the online channel to complete and/or submit their application. The percentage of respondents can also include clients that downloaded electronic application forms and instructions.

Figure 8 – Text version

Proportion of clients using online channels at the Aware stage

Fiscal year	Proportion of CPP clients	Proportion of CPPD clients
2017 to 2018	41%	38%
2019 to 2020	62%	44%
2020 to 2021	66%	50%

Proportion of clients using online channels at the Apply stage

Fiscal year	Proportion of CPP clients	Proportion of CPPD clients
2017 to 2018	30%	23%
2019 to 2020	53%	29%
2020 to 2021	59%	28%

Proportion of clients using online channels at the Follow-up stage

Fiscal year	Proportion of CPP clients	Proportion of CPPD clients
2017 to 2018	30%	22%
2019 to 2020	43%	31%
2020 to 2021	46%	35%

Other online Canada Pension Plan interactions

The expansion and promotion of additional online services was expected to contribute to increased uptake. Performance information indicates that online services experienced increased uptake over the evaluation period (Table 3). Several key informants highlighted specific portions of the CPP applications / CPP SIS initiatives becoming available online that greatly improved client service.

Notable improvements included the ability for clients to apply for the Child Rearing Provision (1.1.3) as part of the CPP Retirement application, the online federal voluntary tax deduction request (1.1.4) and online components for the Children’s Benefit (1.4.1). Other major service improvements noted by key informants were the Document Upload system, as well as the online application informing clients what documents they would need to send in (for those that could not be uploaded).

Achieved volumes and take-up rates for CPP SIS online service initiatives

1.1 Expand and promote online services

Targeted Invitation to Apply (ITA), Tax Info Slips Online, Consent to Communicate, Payment Details and Benefit Attestation. 

• By March 2020, more than 1.3 million Invitation to Apply letters had been sent out
• Between March 2018 and March 2020, increased annual uptake for online Consent to Communicate (15.93%)
• Between September 2017 and March 2020, increased annual uptake for the Tax Information Slips Online (17.10%)
• The total monthly e-service uptake increased by 18.38% between June 2015 to March 2020

1.1.3 Child Rearing Provision

Enabled clients to request the Child Rearing Provision online or during their CPP Retirement online application.

• This initiative did not have the same uptake as other online services. By March 2022, the achieved volume was below the expected volume by 10,133 (expected volume of 31,000)

1.1.4 Voluntary Tax Withhold

Enabled clients to add their Federal Voluntary Tax Deduction during the CPP Retirement online application.

• Between March 2018 and March 2020, Federal Voluntary Tax Deduction requests increased by 29.82%
• The expected volume was 60,000, and by March 2022, the actual volume exceeded expectations by 65%, reaching 99,094

1.1.9 View My Application Status

Enabled clients to view the status of their CPP application through their My Service Canada Account.

• By March 2022, the actual volume was 222,863. No expected volume was set for this functionality

1.4.1 Children’s Benefit (over 18 – School Attendance)

Enabled clients to renew their child benefit online.

• In 2021 to 2022, 10.8% of over-18 child benefit renewals were processed automatically

1.5 Canada Revenue Agency Collaboration

Contributed to improving the client experience by consolidating separate Government of Canada contacts for benefits processing, including Direct Deposit information. Also, by linking clients with their Canada Revenue Agency account.

Client satisfaction

The expansion of online services and streamlined applications was a lever for improving the CPP service experience. This objective was in line with the 2016 Service Strategy vision “to deliver easy to use, seamless, digitally enabled services that put the needs of our clients first” and, more specifically, streamline pension benefits processes.

Results from Client Experience Surveys indicated that the ability to complete steps online was increasingly perceived to make the process easier. In 2017 to 2018, 42% of CPP respondents agreed that completing steps online made the process easier and 62% agreed in 2020 to 2021.

Comparatively, in 2020 to 2021, less than half (40%) of CPPD clients respondents to the client survey agreed that completing steps online made the process easier. Figure 9 illustrates that this marked a significant increase from (2017 to 2018) (29%).

• Source: Client Experience Surveys (2017 to 2018) to (2020 to 2021).

Figure 9 – Text version

Fiscal year	Proportion of CPPD clients
2017 to 2018	29%
2018 to 2019	31%
2019 to 2020	37%
2020 to 2021	40%

However, over the same period, CPP and CPPD respondents to the Client Experience Surveys did not report any significant increases or decreases in their ratings of ease of application for all channels. Still, CPP respondents consistently reported higher ratings than CPPD respondents, as shown in Figure 10.

• Source: Client Experience Surveys (2017 to 2018) to (2020 to 2021).

Figure 10 – Text version

Fiscal year	Proportion of CPP clients	Proportion of CPPD clients
2017 to 2018	88%	57%
2019 to 2019	88%	55%
2019 to 2020	88%	60%
2020 to 2021	85%	57%

The 2020 to 2021 Client Experience Survey reports an increase in CPPD respondents’ satisfaction with client journey service attributes similar to previous years (see Table 2). However, CPPD clients continue to report significantly lower agreement compared to the combined agreement from all client groups (CPP, CPPD, Employment Insurance, OAS / Guaranteed Income Supplement).

Table 2: (2020 to 2021) CPPD client attribute ratings (4 or 5 on a 5-point scale) and percentage point changes from (2019 to 2020)

Satisfaction attributes	2020 to 2021 Rating	Percentage point change from 2019 to 2020
Ease of figuring out eligibility	46%	+ 7
Ease of understanding information about the program	60%	+ 2
Confident application would be processed in a reasonable amount of time	40%	+ 6
Ease of putting together the information needed to apply	44%	+ 1
Ease of following up on application	52%	+ 4
Reasonable duration of the client journey	57%	+ 8
Trust	67%	+ 3
Overall effectiveness	58%	+ 1
Clarity of process	56%	+ 5
Clear what to do if a problem or question came up	60%	- 1

• Source: 2020 to 2021 Client Experience Survey.

The vast majority of key informants indicated that the CPP service experience had improved for clients through the introduction of online applications and other functionalities. Key informants pointed out that CPPD clients could only previously complete certain portions online before being told to move to a paper application; now the whole application was online.

There absolutely was little online presence prior to these initiatives. We went from zero to having it and that was a huge shift. The expected outcome was achieved there.

While opinions were mixed regarding the streamlining of applications and ease of applying, a few key informants mentioned that providing the online option for CPPD clients also improved accessibility (for example, by not forcing clients to visit a Service Canada Centre, by being able to apply at any time of day and save the application to revisit it at a later time).

Client perceptions of being able to move through all the steps smoothly was also relatively stable for CPP and CPPD clients, with CPP clients reporting higher agreement than CPPD clients, as seen in Figure 11. It should be noted that CPP clients reported a decrease in satisfaction with the effectiveness of the process in 2020 to 2021 compared to 2019 to 2020 (85% vs. 80%).

• Source: Client Experience Surveys (2017 to 2018) to (2020 to 2021).

Figure 11 –Text version

Fiscal year	Proportion of CPP clients	Proportion of CPPD clients
2017 to 2018	84%	55%
2019 to 2019	83%	62%
2019 to 2020	85%	57%
2020 to 2021	80%	58%

Client Experience Survey data do not demonstrate any significant changes to the service experience satisfaction of CPP or CPPD clients. CPP clients consistently reported higher satisfaction than CPPD clients. However, both programs reported relatively stable satisfaction scores over the 4 years, as seen in Figure 12.

• Source: Client Experience Surveys (2017 to 2018) to (2020 to 2021).

Figure 12 – Text version

Fiscal year	Proportion of CPP  clients	Proportion of CPPD clients
2017 to 2018	87%	64%
2019 to 2019	87%	62%
2019 to 2020	88%	60%
2020 to 2021	86%	63%

The expansion of online services and streamlined applications were also expected to decrease the time required for clients to complete CPP applications. While no baseline application completion times were established, the evaluation found evidence that both the online CPP and CPPD applications decreased the number of questions or documents that applicants had to fill out.^{Footnote 57}

Understanding Canada Pension Plan Disability client journey difficulties

The increase in online services facilitated access for clients. Yet, evidence shows that CPPD applicants continue to have a lower client satisfaction compared to other ESDC clients.^{Footnote 58}

Multiple barriers or challenges were identified, which could not be generalized. However, based on multivariate analyses from 3 years of survey data (2017 to 2018) to (2019 to 2020) the higher claim denial rates and the longer application journey for CPPD clients was largely responsible for the difference in client satisfaction.^{Footnote 59}

Findings from the 2021 Qualitative Client Experience Research on Service Channel Access with ESDC clients highlighted that difficulties related to the CPPD program were usually encountered during the information-gathering and application stages of the client journey.^{Footnote 60} Some CPPD applicants, typically those with cognitive disabilities, described the information-gathering stage of the process as “overwhelming” or “daunting”.

They attributed these feelings to not understanding what they were being asked to provide or being uncertain that they had provided the correct or sufficient information.

During the application process for CPPD, the most frequent challenges identified in the Qualitative Client Experience Research (2021) related to the length of the application form, as well as the amount and type of information required to complete the form.

For example, many clients experienced frustration, anxiety or confusion when filling out the form.^{Footnote 61}

Movement between channels

The literature review suggests that the preferred client service channel (for example, phone, in-person or online services) can be attributed to the demographic group seeking services, with some population groups at higher risk of being vulnerable, such as seniors and persons with disabilities.^{Footnote 62}

Furthermore, research shows that a service-delivery model that pushes clients towards the online channel may leave certain segments of the population behind because it incorrectly assumes equal access and knowledge of digital services.^{Footnote 63}

These observations are consistent with focus-group findings, indicating that clients generally appreciated the options provided by the different channels. For example, some participants mentioned calling Service Canada agents (when facing difficulties online or not being able to use online services owing to physical disabilities) and preferring to apply in person.^{Footnote 64}

Objective 2: Excellence in performance and results

Has the project improved performance and results by increasing speed of service?

Benefit C: Increased speed of service and reduced waiting times

How the Benefit is planned to be achieved

• Increased processing automation
• Implementation of process improvements
• Implementation of workload management improvements

The initiatives included under Objective 2: Excellence in Performance and Results were designed to increase the speed of service and reduce wait times for CPP clients through increased process automation, process improvements and workload-management improvements.

Increased speed of service and reduced waiting times

The document review found evidence that initiative 2.1.2 Notice of Entitlement contributed to process automation.^{Footnote 65} The initiative was intended to decrease the amount of manual work by reducing the volume of suppressed or rejected Notice of Entitlement letters for CPP and CPPD applicants.^{Footnote 66} This initiative resulted in a 30% decrease in monthly average errors, increased processing consistency and decreased manual work.^{Footnote 67}

Three initiatives were designed to contribute to process improvements in order to support decreased waiting times for CPPD adjudication. Initiatives 2.2.1 and 2.2.2 were implemented nationally following the pilot.   

2.2.1 Dispute Prevention Prototype (Client Engagement Pilot).

• Piloted new client engagement throughout the application process to validate information received, explain decisions and provide other support

2.2.2 Terminal Illness and Grave Conditions Pilot.

• Designed to simplify the initial application by using existing medical information to streamline the process and reduce client efforts to apply for CPPD benefits and provisions

2.2.3 Long-Term Disability Insurers Prototype.

• Piloted the sending of insurers’ medical information at the time of application to streamline the CPPD medical adjudication process to create processing efficiencies to improve client service, while reducing administrative costs

Evidence gathered through document reviews suggests that CPPD Departmental Service Standard indicators for speed of service and wait times were improving.^{Footnote 68} As shown in Figure 13, 2 out of 3 indicators were exceeding service standards in 2021 to 2022, Terminal Illness applications decisions just slightly lower than the national objective of 95%.^{Footnote 69}

• Source: ESDC. Administrative data.

Figure 13 – Text version

• 87% of CPPD application decisions were made within 120 days in 2021 to 2022 (+10% since 2017 to 2018). National objective is 80%
• 96% of Grave Condition application decisions were made within 30 calendar days in 2021 to 2022 (+24% since 2017 to 2018). National objective is 80%
• 92% of Terminal Illness application decisions were made within 5 business days in 2021 to 2022 (+3% since 2017 to 2018). National objective is 95%

Several key informants confirmed the increased speed of service and reduced wait time for terminal illness applications. Key informants highlighted the value of the Terminal Illness and Grave Conditions Pilot given the need to respond to these applicants as promptly as possible.

Speed of service has definitely improved. For gravely ill or terminally ill applicants, those are shorter application processes even in the length of their application. They can do those online and they move along ASAP. The turnaround time can be as little as 2 days if all the information is there. With grave apps, it could be 5 to 7 days.

The document review also found evidence that the CPP SIS 2.2.1 Client Engagement Pilot initiative positively impacted CPPD services by decreasing^{Footnote 70} the following:

• the reconsideration rates, indicating that eligible clients were being granted benefits after their initial application
• the number of appeals, potentially preventing backlogs and excessive wait times for decisions by ensuring that eligible CPPD clients were granted benefits

As shown in Figure 14, satisfaction was rated higher for CPPD respondents who received a follow-up phone call from Service Canada prior to a decision.^{Footnote 71}

Figure 14 - Text version

• In 2020 to 2021, 72% of contacted respondents were satisfied (+18% compared to non-contacted respondents) 
• In 2019 to 2020, 69% of contacted respondents were satisfied (+13% compared to non-contacted respondents) 

In addition, the Client Experience Survey collected data on the perceived speed of service for the CPP and CPPD client journey. The proportion of CPP respondents who agreed (strongly agreeing or agreeing) that the amount of time it took, from gathering information to getting a decision on the application, was reasonable remained steady over time. However, there was a significant increase in CPPD respondents strongly agreeing or agreeing in 2020 to 2021, when compared to 2019 to 2020, as indicated in Figure 15.

• Source: Client Experience Surveys (2017 to 2018) to (2020 to 2021).

Figure 15 – Text version

Fiscal year	Proportion of CPP clients	Proportion of CPPD clients
2017 to 2018	80%	47%
2019 to 2019	83%	49%
2019 to 2020	83%	49%
2020 to 2021	81%	57%

Lastly, the Long-Term Disability Insurers Prototype did not transition to Operations for implementation at the time if this review. Based on the document review,^{Footnote 72} medical insurance information submitted at the time of application did not effectively increase the speed of service despite showing some efficiency. Agents were often required to contact the insurer again in order to request updated medical information owing to extensive processing delays.^{Footnote 73}

Online applications and workload-management improvements

Initiative 2.3 Implement Workload-Management Improvements was expected to increase speed of service through a decreased backlog, age of inventory and key performance indicator achievement for high-priority tasks.

The evaluation did not reveal any data documenting^{Footnote 74} the backlog of items in need of processing by Service Canada agents. Also, no data was located on the number of months of items in its backlog or the rate at which Service Canada agents resolve items in the backlog.

Key informant opinions were mixed about the increase in speed of service and workload-management improvement. One of the key issues raised with the CPPD application was that the applications completed online arrived as PDF files and were not automatically entered into the processing system.^{Footnote 75} Therefore, information still needed to be entered manually.^{Footnote 76}

It [CPPD application] still needs to be processed the same way. Although the information is digital, it does not lessen the processing and analysis that needs to be done. A lot of these applications are not automated. Agents still have to review and analyze the information provided.

Moreover, legislative limitations for clients to upload some supporting documents (for example, copy of original birth, death and marriage certificates) in the CPPD online application was raised as a main pain point, leading to delays in the review of applications while agents waited for mailed documents.

However, many key informants also provided reasons for the increased speed of service and reduced wait times, including:

• aspects of the CPPD application being automated on the back end
• applications being immediately submitted once they were completed instead of having to go through the mail system
• everything already being in electronic form instead of having to be scanned/entered in

A few key informants also noted that fewer disability applications submitted online were being denied or sent back. The reason given was that individuals could not skip questions and were more likely to provide the needed documentation, resulting in reduced wait times overall. To that end, a few key informants clarified that increased speed of service was heavily dependent on the client’s ability to complete the online form correctly and submit the required documentation.

The online application augmented the ability of people to have better and faster service, direct access to sending information and cutting down on the additional time required for citizens to receive benefits.

Objective 3: Excellence in program stewardship

Has the project contributed to long-term CPP program sustainability through reducing operation costs or cost avoidance, and through program-level integrity improvement and risk management?

Benefit D: Long-term and/or program-level integrity improvement and risk management

How the Benefit is planned to be achieved

• implementation of technology security measures
• planning and design of predictive analytics
• implementation of improved decision-making tools

Benefit E: Reduction in future operating costs

How the Benefit is planned to be achieved

Reduction in manual effort as a result of technology, process, and policy changes delivered through CPP SIS initiatives

Objective 3 initiatives (see Appendix B) were designed to contribute to CPP program integrity and risk management. It would also expect to contribute to reduce operating cost through technology security measures, predictive analytics for risk-based processing and the implementation of improved tools to enable decision making.

• The evaluation found some evidence to support that reviewed initiatives contributed to CPP program integrity, risk management and sustainability
• Although the CPP SIS Project Delivery team reported being on track to achieve the estimated cost savings, there was insufficient data to correlate actual savings achieved through the implementation of the full scope of initiatives

Improvements to long-term and/or program-level integrity and risk management

Initiative 3.1 Introduce a Personal Access Code Expiry Date was designed to increase program and client information security. The code aligns with the expiry date of other important documents, such as the Employment Insurance access code, passport, bank credentials, etc. According to the CPP SIS Key Accomplishments Report, over 23,000 inactive personal access codes had been disabled by March 2020, but the document review did not find data on trends over time.^{Footnote 77}

Initiative 3.2 Enhancement of System Access Control and Audit Trail did not have any assigned quantitative benefits, but was intended to improve internal processes. The document review determined that the analysis process brought forward a number of solutions, including:

• efficiencies to improve access to streamlined information and contribute to process consistency
• reduced manual processing and increased automation
• increased focus on high-risk, complex work
• reduced advisory overhead and direct costs
• advanced risk-based compliance opportunities by leveraging ongoing departmental projects^{Footnote 78}

Initiative 3.3 Exploration of Predictive Analytics for both the CPP and CPPD programs was designed to explore opportunities for increased automation, better decision-making and improved client experience. As part of the initiative, the Appeal Management Pilot was found to have improved the efficiency of the appeals management process and was subsequently implemented nationally.^{Footnote 79}

The significance of these studies was used to determine whether predictive analytics provided a feasible and viable opportunity to improve business processes and client service.^{Footnote 80}

Initiatives 3.4.1 CPPD Abbreviated Adjudication Decision Template (Part 1) and 3.4.2 Pensions Knowledge Management Tool were designed to improve internal tools that ultimately contributed to increasing CPP processing accuracy rates.

• Initiative 3.4.1 concluded that a simplified adjudication template could lead to faster decisions for low-risk clients, better quality of analysis clearer written communication and better adherence to policy ^{Footnote 81Footnote 82}
• Initiative 3.4.2 created a single repository for easy-to-understand and updated CPP guidance materials available to regional processing networks^{Footnote 83}

The document review shows that the accuracy of CPP payments continues to exceed the 95% departmental standard, reaching 99.9% in 2020 to 2021.^{Footnote 84}

Reduction in future operating costs

CPP SIS initiatives were intended to reduce operating costs by decreasing the required manual effort through technological, process and policy changes.

Cost efficiency assumptions for calculation

Cost efficiencies were based on the estimated impacts that CPP SIS initiatives would have on client application volumes and processing activities. They included both salaried and non-salaried^{Footnote 85} cost efficiencies.^{Footnote 86}

The expected CPP SIS cost efficiencies from (2018 to 2019) to (2026 to 2027) were $78.5 million.^{Footnote 87}

• 9 initiatives^{Footnote 88} had estimated salary efficiencies totalling $25.5 million^{Footnote 89}
• Another 9 initiatives^{Footnote 90} were bundled together and expected to deliver a 5% efficiency benefit ($19.6 million) as a percentage of CPP program processing spending for combined salary efficiencies
• 5 initiatives^{Footnote 91} were determined to have no material cost efficiencies or unknown cost efficiencies at the outset of the project
• The non-salary-based cost efficiencies were estimated to account for $7.9 million, primarily through avoided imaging fees and medical reports

Indirect cost efficiencies were estimated to account for another $25.6 million based on avoiding future indirect costs for business expertise, corporate salary/non-salary costs and employee benefit plans.^{Footnote 92}

Canada Pension Plan Service Improvement Strategy profile estimated cost efficiencies

As demonstrated in Figure 16, CPP SIS estimated cost savings were largely exceeded as of March 2022,^{Footnote 93} with a positive financial variance of $21.4 million (expected financial benefits of $20.6 million vs. achieved estimated benefits of $41.9 million).^{Footnote 94}

• Source: ESDC. Administrative data.

Figure 16 – Text version

Fiscal year	CPP SIS cost efficiencies estimated prior to implementation (2016) In millions ($)	CPP SIS cost efficiencies estimated at project closure (2022) In millions ($)
2018 to 2019	0.67	16.41
2019 to 2020	5.27	23.49
2020 to 2021	12.19	32.37
2021 to 2022	20.57	41.93
2022 to 2023	30.5	n/a
2023 to 2024	42.07	n/a
2024 to 2025	53.94	n/a
2025 to 2026	66.1	n/a
2026 to 2027	78.49	n/a

Based on the review of financial data up to March 2022, the positive variance resulted from a variety of factors including:

• positive net volumes of e-services uptake owing to early implementation^{Footnote 95}
• salary efficiencies from initiatives^{Footnote 96} not included in the original volume estimate^{Footnote 97}

Canada Pension Plan Service Improvement Strategy profile actual cost savings

There was insufficient data to attribute actual savings related to the implementation of CPP SIS initiatives and validate the efficiency assumptions used to calculate expected savings.

The cost analysis was therefore incomplete owing to limited information on cost per channel, volume of applications by channel (online, paper and in-person) and cost per unit of output. The 2020 Evaluation of Providing Services and Information to Canadians through Service Canada highlighted similar limitations in providing results on cost-effective access to ESDC services and benefits.

On this matter, the 2013 Audit Report on Access to Online Services issued a recommendation for service-delivery departments, including ESDC, to develop and implement standardized approaches and tools aimed at promoting the consistent identification and reporting of service-delivery costs across various channels to support decision making.

While the Department developed a preliminary service-channel costing methodology in 2016 to 2017, the work was never completed nor implemented. 

4.4 To which extent did the Canada Pension Plan Service Improvement Strategy contribute to Canada Pension Plan service-delivery outcomes (“Clients receiving high-quality, timely and efficient services that meet their needs”)

One key commitment of the 2016 ESDC Service Transformation Strategy^{Footnote 98} and driver behind departmental modernization strategies (such as CPP SIS) was to support “clients receiving high-quality, timely and efficient services that meet their needs.” This outcome is also a core ESDC objective and service-excellence commitment.^{Footnote 99}

The evaluation found evidence that the CPP SIS supported the departmental objective and addressed some of the challenges identified in previous audits and evaluations.

Documented client needs

Previous CPP and CPPD evaluations and audits highlighted opportunities and recommendations to improve service quality, timeliness and efficiency in order to meet client needs. The document review identified the following service challenges for CPPD clients:

• applying for CPPD benefits was complicated, time consuming and confusing
• many applicants wanted to know if they were eligible for CPPD prior to pursuing the full application^{Footnote 100}
• clients applying under the Terminal Illness or Grave Condition benefit streams were not receiving a decision within ESDC’s established timeline
• claims of eligible applicants were being denied
• average wait times for clients appealing a benefit decision was over 2 years ^{Footnote 101}

The document review identified the following service challenges for overall CPP Program clients:

• some Canadians living in rural and remote areas, Indigenous people and those with lower levels of education were less able to access the services and information they required
• clients did not feel they had, or could find, sufficient information to make an informed decision regarding when to start their retirement pension. This was particularly the case for clients with disabilities, Indigenous clients, immigrants and clients with lower levels of education/income
• lower-income clients and persons with disabilities found the application process more difficult^{Footnote 102}

Extent to which benefits were reached

Similar to what was noted in previous sections, the CPP SIS supported departmental objectives by providing clients with more timely and efficient services; higher quality client service (for example, more completed forms resulting in better decision-making); and meeting clients needs with access to online services. However, CPPD clients are still experiencing difficulties in accessing services and applying for benefits, which could be attributed to barriers to accessibility (digital and non-digital). Looking ahead and building on activities implemented under the CPP SIS, those barriers should be considered to continue strengthening the delivery of the CPP program.

More specifically, during interviews, some key informants indicated that the CPP SIS improved the timeliness of the application process. They indicated that clients were able to apply directly and submit documents online, as opposed to having to do so by mail, by visiting a Service Canada Centre or by contacting the call centre. This ultimately contributed to a faster turnaround between the client applying and receiving benefits.

People want to be able to go online to complete applications. Perhaps during the pandemic that was even more the case. So I do think it improved client service and arguably allowed us to respond favourably to a critical situation that no one expected.

In addition, some key informants indicated that the CPP SIS contributed to client-service efficiencies. Specific examples of efficiencies included the following:

• online applications could only be submitted once fully completed, eliminating processing delays caused by waiting for additional information
• fewer file appeals through CPPD client engagement
• clients reaching retirement age were provided with everything they needed to apply online
• streamlined adjudication templates for Grave and Terminal Illness CPPD applications helped to meet service standards

Meeting client needs

The literature review suggested that clients expected to be able to engage seamlessly with the government across multiple channels. Regardless of the type of interaction (for example, applying for benefits online, talking to a contact centre agent, meeting an agent in person), clients wished for a tailored experience.^{Footnote 103}

According to the ESDC Customer Experience Qualitative Report (2020 to 2021), the CPPD application process was not always perceived to take people’s disabilities into account, particularly for clients with disabilities related to mental health. Those clients, representing 30% of CPPD respondents, may have a harder time dealing with the stress of completing the application.^{Footnote 104}

As highlighted in the results of the 2020 to 2021 Client Experience Survey and associated qualitative research, digital and non-technological barriers to accessibility could be elements to consider in regard to the higher difficulty and lower satisfaction for CPPD clients.

As shown in Figure 17, one third (36%) of persons with disabilities have difficulty accessing at least one service channel. Therefore, accessibility could require more than one service channel with intersecting client demographics creating unique considerations for service delivery.

• Source: ESDC Centre of Expertise for Accessible Client Service, AccessAbility Playbook

Figure 17 – Text version

Methods	Percentage of persons having access difficulties
At least one method	35.9%
In person	19%
Internet	17.2%
Phone	13.2%
Mail or Fax	8.4%

The future of Employment and Social Development Canada service delivery

According to the 2018 to 2022 Digital Operations Strategic Plan: “The Government of Canada needs to accelerate its digital transformation to meet changing public expectations, optimize value and become nimbler so that it can adapt faster.”^{Footnote 105} Therefore, from 2021 to 2024, the strategic direction will focus on building modern and secure IT systems with a service design that puts users first.^{Footnote 106}

At ESDC, modernization strategies such as the CPP SIS have contributed to increasing access to online services for Canadians. However, as noted by key informants, the full potential of online services could not be reached with current system limitations (in other words, aging technology limiting connections between processing systems and data monitoring for performance measurement). 

More generally, key informants noted the need to continue moving forward, as the CPP SIS was only “step one” in CPP program modernization and there were still remaining gaps.

Now that this project is wrapping up, our forward-looking piece will be standardization and further automation, increasing the online role down the road, looking at future improvements in a transformation space.

5. Conclusions^{Footnote 107}

The evaluation found that the CPP SIS contributed to the achievement of desired outcomes pertaining to the improvement of client service, performance and results and the stewardship of resources.

CPP SIS was the Department's initial concerted effort to improve client service, performance and results, and stewardship of the CPP Program. There is a recognized need for continuous improvements to increase automation and claim processing speed of service for the CPP program.

Implementation and execution

• Findings indicated that the reviewed CPP SIS initiatives CPP were implemented and delivered within expected scope and budget
• Factors that contributed to successful implementation included communication and engagement with the appropriate stakeholders and CPP SIS flexibility regarding the management of its individual initiatives
• Challenges to implementation included conflicting departmental priorities for IT projects causing delays, staff turnover resulting in a loss of expertise, the absence of a dedicated benefit realization team throughout the project leading to performance measurement gaps, and siloed work between teams

Consultation and engagement with clients and external stakeholders

• Client consultations were considered a model to follow on how to include clients early in the development of initiatives
• Engagement and client-experience testing brought insights to better support online uptake, a more streamlined and client-centric CPPD application and process improvements to decrease CPPD adjudication times

Achievement of expected outcomes

Objective 1: Excellence in client service

There is some early evidence indicating that reviewed initiatives expanded the CPP online channel, such as:

• improved CPP service experience by increasing take-up rates for online applications and online services
• year-over-year increase in the number of CPP clients agreeing that the ability to complete steps online made the process easier

Objective 2: Excellence in performance and results

There is some evidence suggesting that initiatives implemented contributed to improved performance and results through increased speed of service and reduced waiting times and adherence to service standards. For instance:

• process-improvement initiatives did appear to contribute to improved speed of service for CPPD application adjudication
• service standards for Pensions and Benefits were almost all exceeded by 2021 to 2022

Objective 3: Excellence in program stewardship

The evidence shows that initiatives implemented contributed to the integrity, risk-management and sustainability of the CPP program to some extent. Notably, the CPP SIS expected estimated cost efficiencies were exceeded by March 2022, with a positive financial variance of $21.4 million.

However, there was a lack of data to determine the CPP SIS impact of decreased manual work, workload-management improvements and actual cost efficiencies.

Looking ahead and building on activities implemented under the CPP-SIS, the following should be considered to continue strengthening the delivery of the CPP program:

• CPP Disability (CPPD) clients were still experiencing difficulties in assessing services and applying for benefits
• continued barriers to accessibility (digital and non digital)

From these main findings, the following observations can be drawn^{Footnote 108}

1. Innovative methods and approaches, such as user-centred design, could contribute to identify and reduce barriers for CPPD clients and vulnerable populations
2. Improved performance metrics on cost savings, productivity tracking and reporting could support to improve project decision-making

Furthermore, the Government of Canada is committed to accelerating its digital transformation to meet changing public expectations.

The 10-year departmental Benefit Delivery Modernization Program will upgrade technology and to expand system accessibility and functionalities. It will leverage the best practices of the CPP SIS and other modernization strategies in order to increase access to self-service and deliver a modern client experience for the OAS, Employment Insurance and CPP programs by 2030.^{Footnote 109}

Appendix A – Canada Pension Plan overview

Overview of CPP benefits - CPP benefit categories description^{Footnote 110}

Retirement pension

Basic benefit available to contributors at age 60; the retirement pension is based on an individual’s contributions to the CPP.

Survivor benefit

Paid to a surviving spouse or common-law partner and includes a benefit for dependent children until age 18, and up to age 25 for those children of the deceased pursuing full-time education.

Death benefit

One-time lump sum payment to the surviving spouse, next of kin or estate of a deceased contributor.

Disability benefit

Paid to contributors aged under 65 who can no longer work owing to a severe and prolonged disability and who meet both contributory and medical eligibility criteria. It includes a benefit for dependent children until age 18, and up to age 25 for those children of a disability beneficiary pursuing full-time education.

Post-retirement benefits

Paid to individuals who are already receiving a CPP retirement pension and who continue to work and make contributions to the CPP.

CPP provisions

The CPP includes 3 provisions (Child Rearing, Pension Sharing and Credit Splitting) that help compensate for periods when individuals may have relatively low or no earnings.

Post-retirement Disability Benefit

Paid to CPP pension beneficiaries aged 60 to 64 who become disabled but do not qualify for the Disability benefit because they have been receiving CPP retirement pension benefits for more than 15 months.

Appendix B – List of initiatives and completion dates

Table 3a: CPP SIS initiatives and implementation dates - Objective 1: Excellence in client service

Initiatives	Description	Ready for use / Implementation date
1.1 Introduce suite of new e-services*	Expand and optimize online services by introducing a suite of new e-services and online applications.	Delivered throughout project
Tax Info Slips Online (TISO) Invitation to Apply (ITA) letters Consent to Communicate Payment Details Benefit Attestation Request for Reconciliation	Building on the success of the online CPP retirement pension application, expand online application functionality and promote these applications and services in an effort to improve take-up. This included: Consent to Communicate functionality allowing clients to view, change and delete the name of an authorized person who has consent to communication with ESDC payment details and benefit attestation increased functionality, allowing clients to view and print payment details including benefit type, gross monthly payment and deductions	Delivered September 2017 Delivered September 2017 Delivered March 2018 Delivered March 2018 Delivered March 2018 Delivered October 2020
1.1.1 CPP Death Application	Introduce online fillable application for death benefits and enable further automation pending policy changes.	Online form completed May 2021 Data integration completed June 2022
1.1.2 CPP Survivor Application	Introduce online fillable application for survivor benefits (including children’s benefits) and enable further automation pending policy changes.	Online form completed May 2021 Data integration completed June 2022
1.1.3 Child Rearing Provision*	Implement an entirely online application process for clients to apply for the CRP as part of the CPP Retirement online application process or as a stand-alone application.	Completed June 2018
1.1.4 Voluntary Tax Withhold*	Enable clients to add or stop their voluntary tax deduction and add their Federal Voluntary Tax Deduction (FVTD) while applying for their CPP retirement pension.	Completed March 2018
1.1.5 Foreign Benefits	Provide clients with the ability to print Foreign Benefit applications and implement an automated process to trigger International Benefit follow-ups.	De-scoped March 2022
1.1.6 Credit Split Application	Introduce the ability to request credit splits online.	Completed October 2021 New initiative added
1.1.7 Pension Sharing Application	Introduce the ability to request pension sharing online and automate application processing.	Completed October 2021
1.1.8 Pensions Third-Party Portal	Establish a secure electronic process for receiving medical documentation from Long-Term Disability insurers, physicians and those acting on a client’s behalf.	Completed March 2021
1.1.9 View Status of CPP and OAS Applications*	Implement real-time application status to enable a client to view the status of their CPP and OAS applications using My Service Canada Account.	Completed October 2017
1.2 Online CPPD Application*	Implement a self-serve, secure, online application process for the CPPD benefit that pre-screens applicants in order to establish contributory eligibility prior to requesting medical documentation.	Completed January 2020
1.3 Enhanced Death Notification Proof of Concept	Implement system changes to ITRDS in order to enable the automatic suspension of a benefit when certified date of death information is received by the Department from the Social Insurance Registry (SIR) database, allowing for further automation of death benefit processing.	Completed March 2021
1.4.1 CPP Children’s Benefit (over 18 – School Attendance) *	Children over 18 in receipt of a DCCB or Orphan’s benefit will be able to submit their proof of attendance at a school or university (ISP1401) through My Service Canada Account. This will include changes to the ITRDS to reinstate suspended benefits.	Completed October 2020
1.4.2 CPP Children’s Benefit	Establish an automated/streamlined process for the annual renewal of the CPP Children’s Benefit that leverages relevant departmental/partner data.	Completed October 2021
1.5 Canada Revenue Agency Collaboration*	Collaboration opportunities with the CRA were explored to focus on areas of collaboration and were geared towards increasing online take up.	Completed June 2019
1.6 Enable Document Upload Solution for Online CPPD Application*	Enable document upload for online CPPD applications.	Completed October 2020 New initiative added

• * Reviewed initiatives – Implemented prior to December 2020

Table 3b: CPP SIS initiatives and implementation dates - Objective 2: Excellence in performance and results

Initiatives	Description	Ready for use / Implementation date
2.1.1 Automate Record of Earnings	Implement processing system enhancements to maximize automation opportunities and reduce high-touch work in the processing network.	De-scoped October 2021
2.1.2 Notice of Entitlement Process Improvement *	Implement Notice of Entitlement process improvements to eliminate residual manual work.	Completed March 2019
2.1.3 Enhance Pension Sharing Process	Address current limitations that require agents to manually input information for 2 CPP client accounts for every application. Address system errors that have caused incorrect benefit rates, denial of applications, etc.	De-scoped October 2021
2.1.4 Improve Payment Exception Processing (PEP)	Modify ITRDS to reduce the number of scenarios that require accounts to be put into PEP status and therefore reduce the effort associated with these accounts.	Partial de-scope March 2022
2.2.1 Dispute Prevention Prototype*	Develop and pilot new client engagement throughout the application process to validate information received, explain decisions and provide other support.	Completed September 2017
2.2.2 Terminal Illness and Grave Conditions Pilot *	Improve business processes to streamline the processing of these applications.	Completed September 2017
2.2.3 Long-term Disability (LTD) Insurers Prototype *	Based on ongoing Phase 1 pilot results, implement opportunities to leverage information obtained from Long Term Disability insurers in order to improve speed of service and decision making.	Completed September 2019
2.3 Implement Workload-Management Improvements*	Building upon the national onboarding of the Pensions Workload System (PWS), enhancement updates to PWS were identified. Updates to the PWS will be implemented to support enhanced reporting capabilities, an archive solution for Work Item Inventory Distribution (WIID) system data and auto creation and distribution of work items.	Completed September 2019

• * Reviewed initiatives – Implemented prior to December 2020

Table 3c: CPP SIS initiatives and implementation dates - Objective 3: Excellence in program stewardship

Initiatives	Description	Ready for use / Implementation date
3.1 Introduce a Personal Access Code Expiry Date*	Add an expiry date to all CPP and OAS PACs as an integrity measure. This will align with the expiry date of other important documents, such as the Employment Insurance access code, passport, bank credentials, etc.	Completed March 2017
3.2 Enhance System Access Control and Audit Trail*	To improve information technology controls and the availability of system audit trails, perform analysis of Office of the Auditor General (OAG) recommendations on system access controls, audit logging and process improvement recommendations.	Completed October 2017
3.3.1 Predictive Analytics CPP*	Based on analysis and recommendations, develop a plan for future implementation of predictive analytic models and risk-based processing systems for CPP processing by identifying and assessing areas of CPP business processes where predictive analytics models and risk-based processing systems could be leveraged.	Completed June 2019
3.3.2 Predictive Analytics CPPD*	A file development pilot, launched in March 2018, using a predictive model to screen initial applications in order to help prioritize the workload and expedite CPPD case processing. Analysis of the results will evaluate enhanced quality of decisions, improved process efficiency and improved client service before recommending national implementation.	Completed June 2019
3.4.1 CPPD Abbreviated Adjudication Decision Template *	Design and implement a new Disability Adjudication Template that will be used across the network when performing the Medical Adjudication.	Completed March 2017
3.4.2 Pensions Knowledge Management Tool*	Implement an electronic tool to serve as an integrated single knowledge management tool for all functional guidance material for Pensions programs that can be referenced by the regional processing network.	Completed September 2019
3.4.3 Automate CPPD Decision Summary	Automate the CPPD Decision Summary.	De-scoped March 2022

• * Reviewed initiatives – Implemented prior to December 2020

Appendix C – Lines of evidence and limitations

Methods

The following studies were used as lines of evidence for this evaluation.

Key informant interviews

A total of 30 semi-structured interviews were conducted with 38 Government of Canada stakeholders to gather views and perspectives on delivery and benefits from people involved in implementing CPP SIS initiatives. Key informants included management members, National Headquarters officers, benefit officers and medical adjudicators in regional processing centres (Atlantic, Ontario and Western Canada and Territories). These key informants provided their feedback on all 4 evaluation questions addressed in this evaluation and were selected based on their experience and knowledge of the CPP SIS. One-hour interviews were conducted with ESDC employees from June 2022 to August 2022. Most interviewees were Service Canada employees.

Administrative data review

Administrative documentation collected during the implementation of the initiatives (2016 to 2022), including departmental operation delivery financial data, was reviewed and analyzed to provide evidence regarding the implementation, delivery and success of the initiatives with respect to expected outcomes. The review covered lessons learned from consultations, individual initiatives, project close-out documents, cost savings and performance measurement from internal databases. The review also used statistics on application volumes per channel and processing speed.

Document review

Internal and external documentation was reviewed to obtain a better understanding of how the CPP SIS was implemented/delivered and gather information on performance and results. The review covered approximately 100 documents from 2016 to 2022 from internal and public web content. Documents included project and initiative descriptions, audit and evaluation reports, project reports, performance-measurement tools and dashboards, strategic documents, unclassified internal communications and presentations, and online content from ESDC’s intranet site and Canada.ca. The document review provided important context for all evaluation questions.

Literature review

This review encompassed literature from 2011 to 2021. The focus of the literature review was to present different approaches used to increase client service excellence and reduce costs related to public service operations. It included peer reviewed academic articles, grey literature and research reports published by third parties, the technology sector and international organizations.

Client Experience Measurement Project: Service Canada’s client experience surveys and qualitative research

Although not a formal line of evidence, results from Service Canada’s Client Experience Surveys were used to inform this evaluation (2018 to 2019, 2019 to 2020 and 2020 to 2021). In addition, the 2020 to 2021 Service Canada Client Experience Survey included 2 qualitative studies used to better understand the service experience of ESDC clients who experienced a barrier or expressed dissatisfaction with their overall experience. Specific results are presented for CPP and CPPD clients. Requests for further information on Client Experience Surveys and qualitative research can be directed to the Client Feedback Centre of Expertise, Citizen Services Branch.

Limitations

Lines of evidence do not include respondents from Quebec or representing pension beneficiaries residing in Quebec as they receive the Quebec Pension Plan rather than the Canada Pension Plan.

No primary source of information was gathered from CPP and CPPD clients to support the evaluation. Within this limitation, an extensive review of Client Experience Surveys (2018 to 2019, 2019 to 2020 and 2020 to 21) and the 2 qualitative research was done to extract results for CPP and CPPD respondents.

While the Department made efforts to develop a service channel costing methodology, this work was suspended owing to competing priorities^{Footnote 111}. The cost analysis is therefore incomplete, given the limited information on cost per channel, application volumes per channel (online, paper and in-person) and cost per unit of output.

Additional notes to quantify participant responses

Table 4: Scale used to report findings for key informant interviews and focus groups from qualitative studies

Term	Definition
Few	Few is used when fewer than 10% of participants gave similar answers. The sentiment of the response was articulated by these participants, but not by other participants.
Several	Severalis used when fewer than 20% of participants gave similar answers.
Some	Some is used when more than 20% but significantly fewer than 50% of participants gave similar answers.
Many	Manyis used when nearly 50% of participants gave similar answers.
A majority	A majority is used when more than 50% but fewer than 75% of participants gave similar answers.
Most	Most is used when more than 75% of participants gave similar answers.
Vast majority	Vast majorityis used when nearly all participants gave similar answers, but several had differing views.
Unanimous, almost all	Unanimous or almost allis used when all participants gave similar answers or when the vast majority of participants gave similar answers and the remaining few declined to comment on the issue in question.