Evaluation of the Canadian Digital Service

Conclusion

CDS effectively implemented an agile digital delivery model based on continuous improvement. Issues were raised about the adequacy of resourcing relative to CDS’s mandate and the lack of understanding of CDS’s role as it relates to other actors in the Government of Canada.

Findings

CDS implemented a digital delivery model inspired by successful international examples and has been adapting to the challenges it encounters. CDS is committed to the Government of Canada Digital Standards: Playbook, such as designing with users, iterating and improving frequently and working in the open by default. CDS has developed innovative hiring practices and has used a revolving cadre of experienced digital practitioners from the public and private sectors on temporary tours of duty.

Documentation and interview evidence show that improvements to service delivery by CDS are based on lessons learned. CDS is now shifting its approach to meet departments where they are by focusing on departmental readiness, as well as on ensuring that all relevant stakeholders within a partner department are involved at the outset of every project.

CDS does not have the authority to mandate change. In light of this, the evaluation found that in order to make progress with partner departments, CDS needs:

• an empowered service owner who champions a multidisciplinary team, helps break down barriers, manages limitations and enables product development
• a department’s ability to stay the course with a product even when there are organizational changes

There is evidence that the CDS model promotes the fast development of digital products; the team routinely assesses and documents lessons from partnerships, and continuously adjusts to different circumstances.

According to documentation, CDS initiated the provision of platform tools in 2019–20. One platform tool that underwent rapid development and uptake is GC Notify, a free, cloud-based tool that departments can adopt to integrate simple email and text notifications into their services.^{Footnote 9} The development of platform tools was appreciated by most interviewees as an important contribution, even those who were most skeptical of the CDS model. Some interviewees even perceived that developing platform tools for the Government of Canada would be more useful than developing digital products in product partnerships, given that the platform tools can be used by various services and by multiple departments and agencies.

The evidence from two case studies and interviews shows that CDS priorities, scope of work and visibility also dramatically shifted as a result of the COVID-19 pandemic. The pandemic accelerated the growth of GC Notify: over 100 government services regularly send updates to Canadians, including appointment reminders, confirmation of submission and requests for information.

Building on GC Notify is Health Canada’s Get Updates on COVID-19, a new email notification service developed by CDS, Service Canada and Health Canada. This email notification service:

• uses GC Notify and a client interface from Service Canada
• was launched in two weeks and, in two months, 50,000 people had signed up to receive over a million notifications

This growth in the number of users highlights the success of GC Notify and indicates likely success for similar platform services. Potential examples of platform services that CDS could offer include tools for scheduling, replacing PDF forms and creating a digital identity for government services.

The pandemic also expanded the need for CDS’s services beyond the federal government. This need was demonstrated by the temporary authorization to use GC Notify to help provincial governments in response to urgent needs related to the COVID-19 pandemic.

CDS was also called upon to develop COVID Alert, a national exposure notification application. Using Bluetooth technology and protecting people’s privacy, COVID Alert lets individuals know if they have been in close contact with another user who tested positive for COVID-19, which can be disclosed via the application using a one-time key. As of April 2021, nine Canadian provinces and territories are using COVID Alert, the application has been downloaded over six million times, and over 28,000 people have entered a one-time key in the application to let other people know that they may have been exposed to COVID-19.

There is a perception among some interviewees that CDS is isolated from the network of actors supporting digital modernization. For example, an interviewee indicated that CDS is not consistently engaged in discussions on digital transformation, modernization or the Chief Information Officer (CIO) Strategy Council. This feedback is meaningful given the view that having CDS participate in these discussions would be beneficial for government.

Several interviewees felt that CDS interacted more with like-minded, early adopters than with those outside this group. Senior leader interviewees also believe that CDS should work with project proposals from smaller departments. However, the document review showed that CDS established partnerships based on government priorities and departmental readiness. CDS has engaged a range of government organizations, such as the Royal Canadian Mounted Police, Health Canada, the Privy Council Office, Employment and Social Development Canada, the Canada Revenue Agency and Veterans Affairs Canada.

Documentation and interviewees indicate that CDS’s ability to achieve its mandate may be inhibited by its resource level, the high expectations placed on it and the complex needs of departments. Some interviewees pointed out that similar entities in other jurisdictions (namely the Government Digital Service in the United Kingdom and 18F in the United States) are larger and have more authoritative mandates than CDS.

Most partner department interviewees and senior leaders were unclear about CDS’s role in the Government of Canada digital space. Several interviewees were unsure whether CDS teams are meant to play a support role as enablers, whether they are set up to be the lead architects of solutions, or whether they can play both roles. Although the CDS role needs clarification, the COVID-19 pandemic has favoured a delivery-first ethos with CDS being the lead architect of solutions.

Conclusion

CDS met the needs and interests of departments to some extent. CDS’s influence is dependant on partners embracing the type of change and modernization that the Government of Canada and CDS seek to instill. While some partner departments are receptive, the conditions required to enable change are not yet systemic.

Findings

The evaluation found there is general consensus that government needs to become more agile and open, and more focused on using iterative and user-centred methods when developing and delivering a service. There is much less understanding within government about how to do this in practice. Interviewees within and outside CDS indicated that CDS’s value lies in showing what change looks like and what is required.

CDS has addressed the needs of partner departments by building prototypes and digital products, and by using product partnerships as a vehicle for capacity building. Although departments can develop products internally or through contractors, half of partner department interviewees indicated that CDS brought unique value through its cutting-edge expertise, speed and agility. Using an alternative would have taken partner departments longer or resulted in a lower-quality product. A few respondents were confident that the products would not have been developed at all without CDS.

Overall, interviewees believe CDS could increase its effectiveness by working more collaboratively and by better understanding the challenges faced by its partners. It is also important to work with departments to enable the conditions for change and to ensure the success of products and services. Even though challenges manifested in COVID-19 initiatives, CDS representatives have acknowledged the issues and efforts are underway to address them.

CDS aims to influence behaviour and culture change through capacity building; however, shifting cultural norms in partner departments is recognized as a challenge. Some of the issues departments face include dealing with legacy systems and new systems, working with existing suppliers and new vendors, and applying an agile model within the traditional hierarchy of government. Key informants acknowledged that, at times, issues can be related to a department’s willingness or ability to change, and whether there is adequate prioritization and support. These issues are outside of CDS control.

Expected outcome

Partner departments in product partnerships increase capacity in modern IT and service design methods.

Conclusion

Product partnerships increased capacity in partner departments; however, sustaining departmental capacity post-partnership remains a challenge.

Finding

According to CDS documentation, building capacity is about “empowering public servants by providing hands-on training, publishing tools and resources, and building communities.”^{Footnote 10} CDS encourages partners to co-locate staff with CDS product teams and helps departments to develop their own multidisciplinary digital delivery teams.

The evaluation found that CDS built capacity directly or indirectly in partner departments in areas such as:

• user-centred research and design
• agile software development and deployment
• open government
• iterative design and delivery processes
• innovative Human Resources practices

All lines of evidence show that CDS has contributed directly or indirectly to building the capacity of its partners. CDS has helped partner departments create teams that mirror their own, which includes new-to-government positions, such as Product Managers. These team members brought digital knowledge back to their departments after working with CDS.

The following are examples of product development that increased capacity in partner departments.

• The EnerGuide application programming interface (API) developed with Natural Resources Canada (NRCan): CDS helped the Office of Energy Efficiency at NRCan adopt new agile, user-centred design approaches and accelerate the transition of its data to the cloud. The API has been reused by other provincial and municipal governments.
• An online service to report cybercrime to the Royal Canadian Mounted Police (RCMP): CDS helped the RCMP build capacity so that the RCMP can eventually assume responsibility for the ongoing maintenance and refinement of a digital cybercrime reporting mechanism.
• Veterans Affairs Canada’s (VAC) benefits finder for veterans and their families: Interviewees noted that capacity was developed among VAC staff in agile, user-centric design and other innovative techniques as a result of the partnership with CDS. VAC assumed responsibility for the product and was able to make improvements (iterative updates) to it more often. VAC’s multidisciplinary team mirrored the CDS team and this mirroring was identified as a success.
• CDS supported the Canada Revenue Agency in prototyping a new tax-filing service that helped create a playbook for conducting real-world testing.

The evaluation found, however, that sustainability in partner departments is a significant challenge to CDS’s capacity-building efforts. Some interviewees from partner departments described instances where they were unable to independently manage their product once CDS ended its involvement because they lacked the capacity to maintain the needed level of user-centric improvements after the end of the partnership.

Evidence from key informants and case studies indicated a need for departments to better prepare themselves before or during the CDS partnership. One department used for a case study created a plan for the post-partnership phase. In doing so, the department was able to manage the product at the time of the handover. This type of plan would enable senior management to better understand and anticipate the resource and change requirements of an ongoing, long-term commitment, and would support informed decisions.

Lastly, another challenge that hinders CDS’s capacity-building efforts in departments is the gap in the current Government of Canada job evaluation standards. Both interviewees and the documentation showed that the current occupational group structure does not define the nature of the work being introduced by CDS, such as design research, service design and product management. This results in employees who do the same job being slotted into different classifications, such as Computer Systems (CS), Information Services (IS), and Economics and Social Science Services (EC), which has implications on recruitment, fair pay, union sensitivities and mobility.

Conclusion

The evidence indicates that CDS products meet user needs. Tension does arise at times between CDS’s implementation of user-centred methods and the established ways of delivering services in some partner departments.

Findings

Interviews and most case study evidence indicate that user-centred design has been the major strength of CDS product development in partnerships. Most interviewees also perceived that user-centred design is needed but greatly lacking within the Government of Canada.
In all partnerships, CDS conducts an intensive discovery research phase (see Figure 1) to:

• define the problem
• explore specific user needs
• identify possible solutions to meeting those needs

The discovery phase entails desktop and design research with users, stakeholders, and departmental staff through interviews, surveys and usability testing. For example, work on Canada Pension Plan disability benefits with Employment and Social Development Canada involved reaching out to different groups of program applicants, health professionals, as well as program staff during the discovery phase. Representatives of the CDS Operations Unit (that is, the policy unit team) and IT support were also engaged.

CDS also deploys various strategies to recruit and engage potential participants for research purposes (for example, keeping interactive bulletin boards open^{Footnote 11} to participants after the end of an interactive session in order to get further insights or using snowball-sampling techniques).

CDS uses an iterative approach throughout the development phase (see Figure 1). Interviewees recognized CDS’s ability to develop tools that meet user needs and noted that products are designed to be user-friendly, while meeting accessibility needs and providing better user experience than comparable tools. One example was Employment and Social Development Canada’s COVID-19 Benefits Finder, which was developed quickly and involved rounds of moderated usability testing, a user survey and iterative user feedback.

An issue that emerged through case studies was balancing departmental requirements and constraints (for example, how to conform to existing approval structures, processes, investments or infrastructure) with the needs of end users, and how to reconcile these requirements and needs when they differ. In one partnership case study, although the discovery phase findings were valuable to the department, CDS was viewed as prioritizing user needs without adequately considering management priorities. The issue was not resolved and resulted in the termination of the partnership.

In another case, user needs versus departmental requirements led to delays that impeded the product’s usefulness. CDS designed a tool in a few weeks using language tested and preferred by users, but which was not compatible with the program language in legislation and in the regulations. These differences meant that program content and departmental approvals had to be re-validated. As a result, users accessing benefits at the beginning had to find information through other sources, which reduced the tool’s impact.

CDS is meant to:

• help departments consider both legitimate departmental constraints and user needs
• ensure that the constraints do not prevent user needs from being met

However, if departments are unwilling to do things differently or if there is no compromise in a partnership, the result may mean retiring, delaying or not releasing a product.