Audit on service provider invoicing controls for the Email Transformation Initiative

Executive summary

What we examined

The objective of this audit was to provide reasonable assurance to Shared Services Canada (SSC) regarding the accuracy of the invoicing reports provided by the service provider of the Email Transformation Initiative (ETI) services.

The scope of the audit included the service provider’s invoicing processes and controls, including all records and systems in support of ETI invoicing and billing from February 1, 2015, to July 31, 2016.

The audit focused on four main areas of the billing process:

• Mailbox reporting
• BlackBerry management services
• Service level targets
• Change request approval process

Why it is important

As part of the whole-of-government approach to information technology, the ETI aims to consolidate and modernize email services by replacing 63 different email systems across 43 partner organizations to one email system. Within this context, a contract was awarded to a service provider; the contract outlines various services, clauses, requirements, statement of work and prices for services.

SSC is dependent on the service provider for data relating to the reports and invoices received and had concerns regarding the accuracy and completeness of the data received to verify the invoices given the high dollar value and impact of the contract. As a result, SSC invoked its right to audit clause in order to identify issues and obtain a more complete view of the accuracy of the service provider’s invoicing data, controls and processes related to the ETI.

What we found

We found that throughout the conduct of this audit, SSC and the service provider were cooperating to find a solution for reporting discrepancies related to mailbox and access management accounts reporting (i.e. the initial step of creating a user identification to allow the generation of a mailbox).

Ongoing dialogue between SSC and the service provider throughout the conduct of this audit, as well as analysis performed by the SSC ETI Team, permitted the identification and correction of BlackBerry reporting discrepancies.

We found that the process used by the service provider for calculating service level targets produced accurate data. However, the retention periods for raw data should be agreed to between SSC and the service provider.

While an effective change management process was found to be in place, SSC and the service provider should define and document what changes are within the scope of continuous email migration work and which changes are required to be presented to the Change Advisory Board for approval.

Sharon Messerschmidt
Chief Audit and Evaluation Executive

Background

Shared Services Canada (SSC) was mandated, as part of the whole-of-government approach to information technology, with the Email Transformation Initiative (ETI) which aims to consolidate and modernize email services by replacing 63 different email systems across 43 partner organizations to one email system. Within this context, a contract was awarded to a service provider for an email service solution, distributed messaging services and mobile device management. The contract includes: applicable contractual requirements, a detailed statement of work and prices for all services.

For all contracting financial transactions, the Financial Administration Act, the Treasury Board Directive on Account Verification and the SSC Financial Management Process together require that:

1. The individuals with delegated authority certify that work has been performed, goods supplied, or services rendered
2. The contract or agreement on terms and conditions have been met including price, quantity and quality
3. The transactions are accurate prior to payment

To perform the ETI contract certification, SSC was dependent on information received from the service provider to verify the accuracy of the data supporting the invoices. Questions were raised by SSC regarding the accuracy of the data received from the service provider and the complexity of the billing and invoicing processes; it was decided that an audit of the service provider’s data integrity controls related to ETI invoices should be completed to provide assurance to SSC on the accuracy of the invoices and its data.

SSC had previously requested that the service provider provide an attestation on their internal controls over the billing process; however, the resulting attestation provided did not meet SSC’s requirements. Given this fact, in addition to other issues in invoicing and billing, and that SSC needed to have assurance over the accuracy of the invoicing data, controls and processes related to ETI; SSC invoked its right to audit the service provider as per the contract.

As part of the established billing and invoicing process, the service provider provides its monthly invoice to SSC, as well as a billing detail file which includes the detailed data for each element of ETI. This includes: detailed information regarding charges to SSC by service item (monthly and one time), amounts owed by SSC to the service provider and raw data to support the invoice. A web-based service delivery portal was created for reporting, account management and service request purposes. Reports provided by the service provider are available to SSC through the portal and used to reconcile the billing detail file to the invoices.

In order to assess the level of service provided to SSC, service level targets were defined in the contract to measure the service provider’s service assurance, service performance and service delivery against specific targets. If a service level target is not achieved the provider must provide a credit to SSC; the method to calculate these credits was identified in the contract and further refinements are documented in a methodology document. The credits can be of high dollar value and SSC was completely dependent on the service provider for data on the achievement of these targets. Because of this, SSC management wanted to ensure the accuracy of data they received regarding actual target data and the calculation of service level targets.

Objective

The objective of this audit was to provide reasonable assurance to the President of SSC regarding the accuracy of the invoicing and billing reports provided by the service provider of Government of Canada email services.

Scope

The scope of the audit included the service provider’s invoicing processes and controls, including all records and systems in support of ETI invoicing and billing from February 1, 2015, to July 31, 2016.

The audit focused on four main areas:

• Mailbox reporting
• BlackBerry management services
• Service Level Targets
• Change request approval process

Methodology

During the conduct of the audit, the Office of Audit and Evaluation:

• Interviewed SSC and service provider employees
• Conducted system walkthroughs
• Reviewed relevant documents, such as the ETI contract, methodology documents and service management guide
• Performed data analysis

Field work for this audit was substantially completed in December 2016.

Statement of conformance

This audit conforms to the Internal Auditing Standards for the Government of Canada, as supported by the results of SSC’s internal audit quality assurance and improvement program.

Sufficient and appropriate procedures were performed and evidence gathered to support the accuracy of the audit conclusion. The audit findings and conclusion were based on a comparison of the conditions that existed as of the date of the audit, against established criteria that were agreed upon with management.

The findings and conclusion are only applicable to the entity examined and for the scope and time period covered by the audit.

Detailed findings and recommendations

Mailbox reporting

We expected information on mailbox reporting provided in the billing detail file (i.e. the detailed data and amounts owed including the numbers and types of mailboxes) and the supporting reports provided to be accurate.

With respect to mailboxes, several issues were noted by SSC regarding the report provided by the service provider to allow SSC to validate data for mailboxes and access management services (i.e. services relating to the initial step of creating a user identification to allow the generation of a mailbox). An analysis from August 2015 to October 2015 showed a high variance between the data in the supporting reports and the billing detail files (ranging from 39% to 78%); therefore, it was not possible to perform a reconciliation between the report and the billing detail file. As a result, SSC was not in agreement with the charges for mailbox and access management services.

We noted that throughout the conduct of this audit, SSC and the service provider were actively cooperating to find a solution to these reporting discrepancies. Change requests were being implemented to include a unique identifier in both the report and the billing detail file that would allow SSC to reconcile the data between both documents. Access management account reporting was also improved by including new attributes for more detailed reporting, as well as by automating the reporting process.

The ETI contract has three different types of mailbox accounts (user, resource and generic). Despite these three different types of mailbox accounts, each having an included allocated number of mailboxes, the service provider did not configure its server to accurately distinguish between types of accounts, and the main database tool was also not connected. This resulted in a discrepancy of what SSC ordered, and what SSC was billed. In many cases, SSC was charged for user accounts instead of the generic accounts ordered.

This situation could exponentially impact on SSC if there is an increase in the volume of accounts generated. Should this accounts conversion continue, SSC would be at risk of surpassing the number of mailboxes per type allowed by the contract and incurring fees erroneously. While SSC and the service provider were actively working together to resolve this conversion issue, during the period of time audited errors could only be corrected manually.

Recommendation 1

The Assistant Deputy Minister, Network and End Users, should, in collaboration with the service provider, develop and implement a solution to resolve the account conversion issue that ensures the type of mailbox (user, resource and generic) is recognized and accounted for correctly.

BlackBerry management services reporting

We expected the information provided in the billing detail file and the supporting reports for BlackBerry Management Services to be accurate.

SSC identified that BlackBerry reporting and data reconciliation between the reports obtained by SSC through the service delivery portal and the billing detail file were inaccurate. These issues included errors relating to personal identification number and phone number duplications, coding, and inconsistencies in the reports. We found the supporting reports for BlackBerry Management Services were reasonably accurate when reconciled to the billing detail file. While they did not align perfectly, a 100% match was not expected due to report timing differences and daily updates to information on BlackBerry Exchange Servers. Additionally, duplications may continue to occur in the future as some users do not delete previous accounts when receiving a new device which causes duplication in both personal identification numbers and phone numbers.

We found that ongoing dialogue between SSC and the service provider throughout the conduct of this audit, as well as the analysis performed by the SSC ETI Team, permitted the identification and correction of these discrepancies. The primary cause of the discrepancies were omissions in the billing detail file. Going forward, an updated service delivery portal report is to be used for a more accurate, and less time consuming, BlackBerry data validation.

Service level targets

We expected the information provided in the billing detail file and the supporting reports for service level targets be accurate.

Based on audit tests performed, we found that the process used by the service provider for calculating service level targets produced accurate data. Data reconciliations between the report provided to SSC and the raw data logs were conducted and no discrepancies were found. Details in the service level targets methodology documentation provided by the service provider were reflected in the walkthroughs performed, and the results obtained, during the audit.

That being said, we found that the raw data logs pertaining to service level targets were retained on the service provider’s servers for three months. An automated weekly deletion cycle in the process deletes any raw data older than three months from the servers.

Industry best practice states that raw data, when maintained internally by an organization, is typically retained for six months and data collected by a service provider pertaining to its own service to a customer is usually retained for one year. Given that any service level target credits that are applied to a billing detail file relate to data from two months prior, there could be an impact (i.e. short turnaround time or dispute) to SSC if the organization wants to review earlier data that may no longer be available.

Recommendation 2

The Assistant Deputy Minister, Network and End Users, should discuss raw data retention periods with the service provider to ensure that it is kept for a reasonable amount of time and meets the needs of the Department.

Change request approval process

We expected the service provider’s change request approval process to have appropriate controls in place and that these controls would be functioning as intended.

SSC was concerned that the service provider implemented a change request that had been rejected by SSC.

The audit team reviewed the documented change request approval process. As part of the change management process, the Change Advisory Board (which included appropriate SSC members) supported the authorization of changes and assisted change management in the assessment and prioritization of changes.

For the specific change request in question, we found no deviations from the documented change request approval process and the change was not implemented after the request was being rejected by SSC. The issue stemmed from a temporary increase in allotments that the service provider implemented in order to minimize operational effects while a permanent solution was found. SSC viewed this as requiring a change management request, while the service provider viewed it as part of the migration work.

Better communication between the service provider and SSC may have prevented this issue from occurring, as each team would have stayed informed of the situation during the change process. Going forward, further discussion and clarification with the service provider could avoid similar issues from occurring.

Recommendation 3

Before any further email migrations, the Assistant Deputy Minister, Network and End Users, should define and document what changes are within the scope of continuous email migration work and which changes are required to be presented to the Change Advisory Board for approval.

Conclusion

Because of reconciliation discrepancies, SSC had concerns about the accuracy of the invoices and detailed billing data received from the service provider. Given the high dollar impact of the contract, and that SSC is solely dependent on the service provider for data relating to invoices, an audit of the service provider’s processes on reporting and controls related to ETI invoices was requested to provide assurance to SSC management on the accuracy of the data used for detailed billing and invoicing. The audit focused on four main areas of the billing process: Mailbox reporting; BlackBerry Management Services; service level targets; and Change request approval process.

We found that throughout the conduct of this audit, SSC and the service provider were cooperating to find a solution to reporting discrepancies related to mailbox and access management reporting. If there is an increase in the volume of accounts generated, SSC could be at risk of surpassing the mailbox allocations and incurring fees erroneously if the servers inadvertently change generic accounts to user accounts.

Overall we found that the process used by the service provider for calculating service level targets produced accurate data. Data reconciliations between the report provided to SSC and the raw data logs were performed and resulted in no discrepancies. We noted that data retention periods were less than the industry best practice of six months to a year. This could have an operational impact for data reconciliation given the short turnaround time provided to SSC.

The service provider has a documented change management process, which includes appropriate SSC membership, and operates and functions as an appropriate control as intended. However, there was a misunderstanding regarding which actions may fall under continuous migration work and those that are required to go through the change request process. Further clarification and communication with the service provider on this issue would help clear the various interpretations of what falls under continuous migration work.

Audit recommendations should be addressed before any further email migrations to avoid a recurrence of issues identified in this report, along with potential additional charges and work on the part of SSC.

Management response and action plans

Recommendation 1

The Assistant Deputy Minister, Network and End Users, should, in collaboration with the service provider, develop and implement a solution to resolve the account conversion issue that ensures the type of mailbox (user, resource and generic) is recognized and accounted for correctly.

Management action plan	Position responsible for action	Completion date
The established service desk and problem management processes were followed to identify and diagnose this issue. It was subsequently reviewed at the weekly operations meetings held between SSC Email Operations and the service provider, where an operational Change Request was generated by the service provider. The Change Request was tested and approved by SSC Email Operations and implemented on December 8, 2016.	Director General, Email Services Sector	December 8, 2016

Recommendation 2

The Assistant Deputy Minister, Network and End Users, should discuss raw data retention periods with the service provider to ensure that it is kept for a reasonable amount of time and meets the needs of the Department.

Management action plan	Position responsible for action	Completion date
SSC has confirmed to the service provider the need to retain raw billing data in order to verify their invoices, and accordingly has sent the service provider a Design Change Request to implement data retention (DCR-325) which will result in a contract change. The service provider was to have provided a cost and implementation estimate for SSC’s review.	Director General, Email Services Sector	June 2018

Recommendation 3

Before any further email migrations, the Assistant Deputy Minister, Network and End Users, should define and document what changes are within the scope of continuous email migration work and which changes are required to be presented to the Change Advisory Board for approval.

Management action plan	Position responsible for action	Completion date
The "Service Management Guide", has been updated in detail to describe all types of changes, their categorization, and the approvals required for each. SSC and the Vendor have regular joint weekly Change Advisory Board meetings where all changes are reviewed and if appropriate, approved. SSC holds the veto power in this Change Advisory Board. These processes have been diligently followed since their creation and documentation. Additionally, the "Service Manager Handbook-Change Management" is updated whenever a Normal Change is being considered for identification as a Standard Change. The "Service Manager Handbook-Change Management" not only documents how this is managed, but also acts as the "catalogue" to document each and every Normal Change that has been promoted to Standard Change categorization. The revision history of the document is also updated accordingly.	Director General, Email Services Sector	March 22, 2017

Annex A – Audit criteria

The following audit criteria were used in the conduct of this audit:

1. The information provided in the billing detail file and the supporting reports for mailbox reporting is accurate
2. The information provided in the billing detail file and the supporting reports for Blackberry Management Services is accurate
3. The information provided in the billing detail file and the supporting reports for service level targets is accurate
4. The service provider’s change request approval process has the appropriate controls in place and is functioning