Language selection

Government of Canada / Gouvernement du Canada

Search

Sign in

Sign in

Review of Impacts of the Government's Secure Channel on CRA Program Delivery

Corporate Audit and Evaluation Branch
November 2006

Table of Contents

Main Messages

From a technical perspective, we have concluded that Secure Channel Common Registration Services is operating effectively in terms of system stability, availability to users, and response times for processing user transactions. We have also concluded that it is providing an appropriate level of security for the transmission and retrieval of private information. As a result, we found no significant threat to the delivery of the CRA program in terms of either on-going service to taxpayers or to revenue collection.

We did find, however, that the registration-authentication processes, including CRA's own out-of-band authentication process, are having a significant adverse effect on the use of My Account. These new processes are more complex for users wishing to access My Account for the first time and result in more that 50% of potential users abandoning their efforts to access the service. There also has been a significant number of logins that do not result in visits to my account. Given that the level of security was found to be appropriate, CRA should explore alternative processes that provide the same level of protection but which are less onerous for the users. This would allow CRA to maintain the level of privacy that currently exists yet minimize the number of potential users who abandon their efforts to access My Account.

While the governance structure has not yet caused any significant operational problems for the CRA, decisions being made with respect the operation of the system could represent a risk to the delivery of the CRA program if not closely managed by the Agency. Further, CRA has concerns with respect to decisions being made and the lack of influence it has within the committee structure. Material concerns should be formally documented and raised to the right governing body or authorities for decision.

We also found that the new funding model for SC-CRS is a significant stumbling block for CRA's future plans for use of SC-CRS. Recognizing that the proposed cost recovery will drive the Agency to minimize the number of services offered through SC-CRS, CRA needs to fully explore a full costing of alternative approaches to SC-CRS to determine real cost savings before making alternate arrangements for delivery of e-services. If CRA is to continue its participation in SC-CRS in a cost recovery environment, it needs to take steps to ensure that the funding model is further developed to ensure clarity and equity in cost allocation and should ensure that decisions to provide services through SC-CRS take into account cost-effectiveness and are justified against business needs.

We found that CRA has no clear service channel delivery strategy that includes goals and objectives for participation in SC-CRS. CRA should develop an overall service channel strategy considering the Agency's 2010 vision for expanding CRA services. As part of the development of the strategy, it should review and rationalize the business requirements to be offered through SC-CRS to ensure they are balanced against costs, ease of use, and level of security required.

Finally, we found that performance measurement and reporting are underdeveloped and fragmented and do not provide a clear picture of performance. CRA needs to develop and implement a comprehensive performance management framework with outcome indicators for CRA services on SC-CRS and re-negotiate the Service Level Agreement with PWGSC to satisfy CRA's needs for SC-CRS metrics.

Introduction

This report presents the findings of a review of impacts of the Government of Canada's (GOC) Secure Channel (SC) epass Common Registration Service (CRS) on delivery of the Canada Revenue Agency (CRA) program. This study was undertaken to assess the validity of concerns that some aspects of SC-CRS may be negatively impacting delivery of the CRA program, and to provide information to guide the CRA with respect to its relationship with this GOC initiative. The study began in February 2006 and was completed in June 2006. It was recognized from the outset that the timelines stipulated for the study posed a significant challenge. As such, the content of this report is a focused on the main problems for CRA program delivery that is attributable to participation in the GOC-SC-CRS and it does not look at all elements of SC.

Background

SC is a GOC initiative to establish a government-wide infrastructure that provides network services for federal departments and agencies supported by security, registration and authentication services that enable them to deliver their most commonly used services on-line. It is intended to reduce overall operational and maintenance costs through the use of a common infrastructure. A key component of the SC is the generation of an epass through a common registration service that ensures secure access and protects privacy during and after on-line transactions.

The CRA has played an important role in SC and launched the first GOC application in September 2002, Address Changes On-Line (ACO). In February 2005, CRA increased the services offered through SC by adding a suite of services that allows individual tax filers to view and manage their personal income tax accounts under a My Account portal for individuals. In February 2006, CRA implemented Represent a Client, which provides authorized representatives with secure and controlled online access to their clients' tax information. Future plans exist to add to this suite of services for individuals as well as creating a My CRA Business Account portal.

It is worthy to note that the CRA has been providing electronic services for many years prior to SC-CRS implementation, most of which continue to operate outside of this initiative. Some of these include: EFILE, NETFILE, TELEFILE (for T1) and GST/HST TELEFILE.

Methodologies

The main methodologies included extensive document and literature review, review of special studies undertaken by CRA branches, interviews with CRA senior officials, managers and staff as well as lead representatives from Treasury Board Secretariat, (TBS) Public Works and Government Services Canada (PWGSC) and the Communications Security Establishment. A one-day ‘points of discovery' session was also held with key stakeholders from the Information Technology, (ITB), and Assessment and Benefits, (ABSB), to validate the information being used for the analysis.

Findings

Many obstacles were faced during the study that significantly impacted the robustness of the findings and recommendations. The major obstacles were inconsistent and insufficient reporting of performance information over the duration of the implementation of My Account, undated project documentation, and external sensitivities about the study that prevented the evaluation team from conducting individual interviews with various members of other federal departments who are managing/participating in SC-CRS. The short timeline for the study permitted only a limited review of a considerable amount of complex information. While the findings are based on the best information available, and on broad considerations of obvious changes and trends, we were not able to fully assess problems with SC-CRS or validate concerns raised from internal interviews.

Overall, we found that from a technical point of view SC-CRS is now running effectively in terms of stability, availability, and response times. Furthermore, we found no significant threat to the delivery of the CRA program in terms of either on-going service to taxpayers or threats to revenue collection. We also found that there have been no security breaches and that the level of security provided by the existing process is appropriate for the most sensitive information.

The main areas of concern identified during the study were:

  1. My Account Take-up
  2. Governance
  3. Performance Measurement
  4. Cost and Alternatives
  5. Service Channel Strategy

1. My Account Take-up

Registration-authentication and login processes have adversely affected take-up of My Account

Prior to February 2005, My Account was accessed through a CRA authentication solution and there was no registration or activation process in place. Migration to SC-CRS introduced a process that requires users to register for an epass by establishing a user ID, password and recovery questions. Authentication is the responsibility of each government department and agency. CRA's authentication process includes providing shared secrets. At the time of conversion to SC-CRS, the CRA implemented an out-of-band process to complement its on-line authentication. This requires the user to wait up to 5 days for an activation code to be sent to their home address through the mail. Users must then login to SC-CRS and enter their activation code within prescribed time frames before they are able to access My Account.

It is difficult to determine definitively the full impact the migration to SC-CRS has had on My Account. Both the My Account and SC-CRS applications have changed substantially as has the methodology used to measure the use of My Account. While performance data was collected both prior to and after conversion to SC-CRS, the data is scattered and not directly comparable. As a result, there is lack of confidence in some aspects of the data.

Overall, however, we found that the actual usage of My Account (web page visits) declined with the implementation of SC-CRS and may not have yet regained the losses. Comparing login statistics for the key months of March and April, we found that logins immediately after the implementation of SC-CRS dropped by 11.5% from 1.116 million in 2004 to .988 million in 2005. For the year 2006, the data suggests that the loss has been recouped as the number of logins increased to 1.114 million for March and April of this year. This recovery may be attributable to a new release of SC-CRS in February 2006 that was intended to improve the user experience.

While on first glance it would appear that we are back to pre-SC levels, other data call that conclusion into question. Since SC-CRS implementation, there has been a significant number of logins that did not result in visits to My Account. To access the My Account Welcome Page, the user must navigate through 3 different web pages before arriving at My Account. CRA officials believe that users end the process for a variety of reasons such as being interrupted or wandering-off to other web-links prior to entering their My Account. The data suggests that, for whatever reason, the drop-off rate during the login process is more than 30%. This has resulted in the number of logins to SC-CRS being considerably higher than the number of actual visits to My Account. For the months of March and April 2006 the drop-off rate was 36% as there were only .731 million visits to My Account. This is significantly lower than pre-implementation visits not withstanding additional services being available. There is, however, a lack of full confidence in the data that supports the 30% drop (performance reporting is discussed later in the report).

We also found that in 2005/06, the length and complexity of the registration-authentication process, which includes the out-of-band process, resulted in more than half of would-be registrants failing to complete the process. Of the 667,617 users attempting to register, 20% abandoned their efforts during the combined CRS/CRA registration-authentication process. This may be due to the complexity of the process, insufficient data available to the user to complete the process, or diversions to other sites. An additional 32% dropped out during the CRA specific activation requirements for its out-of-band process. This may be due to users being unaccustomed to the five-day delay resulting from the mail out of their activation code or to the requirement for them to complete the registration process by activating their epass within 35-50 days.

This suggests that the population of potential users of My Account is limited by the additional requirements associated with the SC-CRS and CRA out-of-band processes. The end result is that many potential users abandon their desire to use My Account because of both the SC and additional CRA security processes. Had all of these users been successful in establishing access to My Account, it is likely that the number of visits would have increased over the last two years rather than having decreased.

TBS and PWGSC believe that CRA has overly complicated the process by introducing the additional out-of-band process. It should be noted, however, that the need for an out-of-band process was identified by the Privacy Commissioner in a letter to TBS in December 2002 as there were concerns that the CRA shared secret information method of authentication, in itself, did not adequately address risks to privacy. The CRA agreed that there was a business requirement to have a strong authentication process in order to provide assurance that the CRA is communicating with the identified tax filer and to minimize the possibility of unauthorized individuals accessing sensitive information. Expert opinion supports CRA's decision in this regard. It was determined that by adding the out-of-band process, CRA's process is more stringent than the basic SC-CRS, and provides an important layer of assurance that is equivalent to Internet banking. Further, the out-of-band process offers protection against “man-in-the middle attack” and focuses on preventing identify theft.

While the resulting registration-authentication process may be appropriate in the SC-CRS environment, it is complex, time consuming, and is contributing to the CRA losing more than 50% of potential users. It is noted that alternatives being explored by the CRA include the different registration-authentication requirements, but including the out-of-band process, and will likely result in a continuation of a significant drop-off rate of potential users during the process. To minimize the number of users who drop out during the process, the CRA, when exploring alternatives, should determine whether there are other options available that offer the same level of protection but that minimize the burden on the user.

It was noted during our study that not all of the services that were expected to be offered through SC-CRS would necessarily require the CRA's out-of-band process, e.g. Netfile. It is recognized that CRA is currently reviewing its e-service delivery options because of cost considerations regarding SC-CRS. During this review, CRA needs to review the current applications utilizing SC-CRS, as well as plans for future applications, to determine if there are services that could be serviced by other methods of authentication.

2. Governance

There is no evidence that the governance structure has caused any significant operational problems for the CRA

The SC governance structure provides various levels of oversight and decision-making for managing this government-wide initiative. The principal governance bodies are the Information Management Board (IMB) and the SC Management Board (SCMB). The IMB is chaired by TBS and oversees the SC at the enterprise level setting overall policies, strategies and priorities. The SCMB, which is co-chaired by TBS and PWGSC focuses on the operational level and oversees activities of the various shared services on the SC. CRA is represented on these committees at the Assistant Commissioner and Deputy Assistant Commissioner levels respectively. The governance structure adopted for SC is consistent with industry practices for managing enterprise wide technology infrastructures.

We found no evidence of any significant problems having actually occurred for the CRA program because of the SC governance structure. Some incidents occurred related to release management with potentially significant consequences for interruption of service for My Account but were resolved without incident. New processes were put into place by PWGSC to lessen risks of these incidents reoccurring but Senior CRA officials are still concerned about the potential for serious consequences. Of particular concern is the lack of enforcement of standards for the introduction of new services or modifications by other departments/agencies, which, if left unchecked, could strain the common infrastructure and diminish effective delivery of CRA applications. It is uncertain whether this is because PWGSC lacks the authority to compel compliance or because PWGSC is not exercising its authority. While the incidents that have occurred could be related to growing pains for system development and release management, it is certainly in the interest of CRA to continue close monitoring of SC-CRS plans for new and modified services to ensure that proper protocols are followed and that threats to CRA services are minimized.

Also of concern to agency management is the lack of influence that CRA has in the SC decision-making process. On the governance committees, CRA's 2 votes out of a total of 14 do not take into consideration the volumes of CRA business on SC-CRS. More importantly, however, CRA management note that voting rarely ever takes place and decisions seem to have been made in advance of meetings by PWGSC and TBS thereby marginalizing the influence of the committee. There is also a strong sentiment that Committee members are not given sufficient time to prepare for deliberations and that meetings are held more to inform than to involve committee members in the process. This compromises CRA's ability to protect its business needs by potentially subordinating the criticality of CRA business requirements to the diverse needs of other SC-CRS programs.

Restricted access to external sources of information and to other government departments offering services on the SC-CRS prevented us from fully assessing the validity and materiality of CRA's concerns about the governance structure. It is, however, unclear whether the CRA has formally documented its concerns or has officially raised and escalated them through the committee structure for resolution. If CRA's objections to aspects of the governance structure and/or its operation are material, it is incumbent on the CRA to formally document them and bring them forward for resolution.

3. Performance Measurement

Performance measurement and reporting are underdeveloped and fragmented and do not provide a clear picture of performance

The lack of adequate performance measurement information is resulting in inconsistent reporting on the results being achieved by SC-CRS and My Account. CRA relies on PWGSC, as owner of SC, to provide data related to technical performance including availability and response time. CRA generates its own information with respect to the volume of visits to its web site, including visits to My Account, information on authentication, registration and logins for epasses.

CRA officials expressed concerns that the performance measurement by PWGSC for SC-CRS is underdeveloped for their needs. This has contributed to a lack of confidence by the CRA on the technical performance of SC-CRS. A review of the draft Service Level Agreement (SLA) revealed there is limited performance reporting required. Furthermore, we found that there are differing definitions for, and understanding of, key elements of performance reporting such as system availability. While it is uncertain whether the current PWGSC reporting satisfies all of the reporting requirements of the SLA, we found the reports were not received within the time frames specified in the SLA. We also found that PWGSC information is unreliable with respect to the reporting of the number of epasses generated by the CRA. While CRA has brought performance measurement issues to the attention of PWGSC, we were advised that further development of performance measurement information has been delayed because of more pressing demands on PWGSC for improvements to the system.

With respect to internal performance measurement and reporting, CRA collects a variety of data on My Account activities. There is, however, no comprehensive performance management framework that clearly sets out goals, objectives and targets. As a result, CRA management has neither clearly set out expectations of what it is expecting to achieve nor performance measures that adequately demonstrate the results it is achieving. At the operational level, performance measurement and reporting is fragmented and carried out independently within three different branches. Generally, performance measurement has been developed on an “as needed basis” and is still evolving with the development of web metrics for the whole CRA website. Currently, ITB reports on technical performance, based on PWGSC data as well as CRA's own system monitoring reports. ABSB uses the report entitled “AMS Statistics for Accessing My Account Flow” to manage authentication and registration and relies on data provided by the Taxpayer Services and Debt Management Branch (TSDMB) to assess the usage of My Account. The performance data is not easily reconciled, and there are questions concerning accuracy or validity of the measurements. As well, performance has been difficult to track because terminology and indicators have changed over time. ABSB recognizes the need to consolidate the statistics for registration, authentication, logins and visits to My Account to provide a better overall view of performance. As a result, significant measures, such as end-to-end response time to measure the complete user experience, are not being tracked and CRA does not have adequate measurement of the results that it is achieving. We have not seen a plan to address this need.

As a result of the deficiencies in both PWGSC and CRA performance measurement, there is no clear picture of how well SC-CRS and My Account are achieving their objectives. Furthermore, CRA management has received inconsistent reporting on issues such as numbers of logins and visits to my account. In order to address the deficiencies, CRA should develop a comprehensive performance management framework for its electronic services on SC-CRS. At a minimum, the framework should clearly define goals and objectives, key performance indicators, and targets. In addition, if CRA requires specific performance measurements and reports with respect to PWGSC metrics to adequately manage its program and assess achievement of objectives, the requirements should be negotiated through the SLA.

4. Cost and Alternatives

Cost is a significant stumbling block for CRA's participation in SC-CRS

Although cost was not the focus when we began the study, it quickly emerged as a significant concern for the CRA and could have an impact on the extent to which the Agency participates in SC-CRS in the future. When the GOC SC initiative began, the Public Key Infrastructure (PKI) was acknowledged to be an expensive solution but was the TBS chosen alternative that met the requirements for the Government On-line (GOL) initiative. While cost has always been a concern, it was anticipated that, as a result of ecomonies of scale, it would end up being a cost-effective service delivery channel.

TBS has now proposed a change that would move from the current centrally funded model to a full cost recovery model for departments and agencies that use SC-CRS services. Cost recovery from departments and agencies with some centralized subsidization will begin in fiscal year 2007/08 with full cost recovery by 2008/09.

The annual cost of SC-CRS infrastructure is now estimated to be in the range of $47-54 million per year for the next seven years. TBS has asked the CRA to commit to projections to support the costing model for the next seven years. The projected costs to the Agency are significant and are based on a transaction cost of $1.15 per login plus costs associated with activities such as testing and help desk support. It is estimated by CRA that the total costs charged by TBS will be between $7.5 million to $14.2 million per fiscal year or $1.28 and $1.60 per transaction. Full cost for the seven-year period are estimated to be approximately $70-76 million. It should be noted that this only represents the costs charged by PWGSC and does not include the costs incurred by CRA to support its SC-CRS operations.

It should also be noted that while the current funding model is based on a transaction charge, the definition of a transaction remains outstanding. Should the definition be based on CRA logins, the Agency could face additional costs associated with those who use its site to login but who never reach My Account for whatever reason.

Alternate technologies exist that may meet CRA business requirements taking into account risks, usability and costs. These options range from an SSL to a PKI solution. Preliminary costing of these alternatives undertaken by ITB indicates that the CRA might be able to meet its own business requirements at a substantially lower cost than those projected for SC-CRS. Based on the initial CRA analysis, transaction costs could range from $0.16 to $0.68 resulting in an estimated annual CRA cost between $1.0 million to $4.5 million. The option preferred by both ITB and ABSB is the in-house development of an SSL-based solution, which is a different solution than SC-CRS. Initial cost estimates for this solution are $3 million for development and $1 million per year in operating costs. Estimated cost per login for this solution in an operating environment is 16 cents.

These initial estimates are, however, very preliminary, being described by ITB as “class Z – feasibility level” estimates, and do not fully assess all associated costs of developing and implementing an in-house solution. We understand that these estimates have yet to be reviewed by ABSB and do not include any costs that might be incurred by them or any other branch in the Agency. As a result, these estimates are understated and are only useful in estimating ‘order of magnitude' of the costs. CRA must undertake a full, detailed assessment to ensure the extent of the estimated cost differences between the SC-CRS solution and alternative CRA solutions are accurate prior to seriously considering alternatives. This assessment must take into consideration business requirements, usability, security, and privacy as well as an appropriate transition strategy.

Since the initiation of this study, TBS has initiated it's own study to explore concerns around cost effectiveness and sustainability. The terms of reference for the ITB study acknowledge that SC was based on concepts, policies, and technology solutions from ten years ago. The study, scheduled for completion by June 2006, is intended to examine whether the business requirements, policy, and standards that led to the current solution is preventing cost competitiveness in providing authentication. As the results of this study could impact the costing model that will eventually be in place for SC-CRS and perhaps even its fundamental underpinnings, the CRA should consider these results when assessing CRA alternatives.

It should be noted that while the funding model is important in today's environment, it could also have an impact on the achievement of the Agency 2010 vision. If the CRA is to be successful in achieving its vision, costs to clients for CRA services must be reasonable. Current cost estimates associated with the SC-CRS have already had a negative impact on discussions with one province with respect to its use of CRA services. CRA's decision on the future of e-service delivery, therefore, must be carefully considered to ensure that it fully supports achievement of Agency 2010 objectives.

5. Service Channel Strategy

No overall Service Channel Strategy exists

In assessing the impact of SC-CRS on CRA program delivery and Agency 2010, we found no documented evidence of a SC-CRS delivery strategy or an overall CRA service channel strategy to guide the Agency in meeting its objective to provide effective, efficient and responsive delivery of CRA programs and services. However, it is understood that the CRA's strategy for several years has been to deliver affordable and accessible client services, and that enhancing electronic service options was integral to achieving this goal. Underpinning this was the assumption that electronic service would be a cost effective service delivery channel.

From the beginning, the CRA has been an enthusistic supporter of the GOL's transformation of service intiaitive, the purpose of which is to provide alternate service options for clients. CRA's initial decisions to migrate service to the SC-CRS initiative was based more on supporting the GOC's citizen-centred vision of providing a single login to a variety of services rather than on program specific delivery, security requirements or cost effectiveness considerations. As a result, all applications within My Account were migrated to SC-CRS and have the same level of security. The complex registration process associated with SC-CRS may be discouraging participation in My Account and driving clients to more expensive, traditional channels for services that could be provided more efficiently through e-services which have appropriate security requirements.

From a service perspective, it is of a value to Canadians for the CRA to offer multiple service delivery channels in order to encourage voluntary compliance. From an efficiency perspective, the CRA must provide service channels that address these needs with a reasonable cost. As such there is a need now for an overall service channel strategy for the CRA. This strategy will be a key requirement in determining the direction that the CRA should take with the SC-CRS service channel. It should ensure that business requirements of SC-CRS are balanced against costs, ease of use, and level of security required. It should also be taken into consideration that e-services are not always the most cost-effective service channel. Furthermore, the use of SC-CRS for services such as Netfile and My Business Account should be rationalized.

The Corporate Strategies and Business Development Branch (CSBDB) is currently developing a Service Delivery Strategy to provide a more coordinated approach to service delivery. The branch is examining the way CRA interfaces with taxpayers and benefit recipients with a focus on the use of, and balance among, various service channels and technology enablers. Development of the strategy should take into account the points noted above.

Conclusions and Recommendations

We have concluded that SC-CRS is operating effectively in terms of system stability, availability to users, and response times for processing user transactions and that it is providing an appropriate level of authentication and security for the transmission and retrieval of private information. As a result, we found no significant threat to the delivery of the CRA program in terms of either on-going service to taxpayers or to revenue collection.

There are, however, some items that the need attention by the CRA as reflected in the following recommendations:

Management Response

Management accepts the conclusions and recommendations of the review. The CRA's decision to migrate service to the SC-CRS initiative was based on supporting the Government of Canada's citizen-centred vision of providing a single login to a variety of services rather than on program specific considerations. The Public Key Infrastructure (PKI) was the TBS chosen approach that met the requirements for the Government On-line initiative. The CRA will continue to support the SC GOC initiative as it is operating effectively in terms of service to Canadians and it provides security for the transmission and retrieval of private information.

The partner relationships within the Secure Channel will enter a new phase upon the signing of the Long Term Contract with the service provider. The CRA will work with the PWGSC and other partners to establish an appropriate costing and governance model to address all material concerns found in the evaluation. PWGSC is in the process of redrafting the Service Agreement (SA) with CRA to cover specific SLA's for each service.

Page details

Date modified: