Evaluation of the Consumer Services Centre 
Final evaluation report

Prepared for:

Financial Consumer Agency of Canada

Prepared by:

Alison Taylor
Performance Solutions Inc.
79 Kenilworth Street
Ottawa, ON K1Y 3Y6

E-mail: Alison.Taylor@live.com

Date: November, 2023

Information contained in this publication or product may be reproduced, in part or in whole, and by any means, for personal or public non-commercial purposes without charge or further permission, unless otherwise specified. Commercial reproduction and distribution are prohibited except with written permission from the Financial Consumer Agency of Canada.

For more information, contact:

Financial Consumer Agency of Canada

427 Laurier Ave. West

Ottawa ON K1R 1B9

www.canada.ca/en/financial-consumer-agency

Cat. No. FC5-91/1-2024E-PDF (Electronic PDF, English)

ISBN 978-0-660-72889-6

© His Majesty the King in Right of Canada, as represented by the Minister of Finance Canada, July 2024.

Aussi disponible en français sous le titre : Évaluation du Centre des services aux consommateurs - Rapport d'évaluation final

Executive summary

For the Financial Consumer Agency of Canada (FCAC) to deliver on its priorities, it requires efficient and timely support from all its branches and services. The goal of the evaluation is to provide the Agency with strategically focused, objective, evidence-based information on the Consumer Services Centre (CSC). The objectives of the evaluation are to assess the CSC’s current role and function and identify opportunities to maximize its contributions.^{Footnote 1}  Specifically, the evaluation assessed the extent to which:

1. The CSC supports the priorities and strategic goals of the FCAC;
2. Effective and efficient services are being provided to financial consumers.

Alignment to FCAC Strategic Goals

In theory, the CSC aligns well with FCAC’s Departmental Results Framework (DRF) Strategic Outcome to protect financial consumers. The CSC also aligns well with the Business Plan strategic goals to "Be the authoritative source of Canadian financial consumer protection information” and “strengthen the financial literacy of Canadians for an increasingly digital world".^{Footnote 2}  The CSC provision of services is also aligned with federal policies, specifically the TB Policy on Service and Digital which is rooted in the principles of client centricity and a culture of service management excellence. However, in practice, corporate performance and planning documents including the Departmental Results Framework (DRF), Business Plans and Annual Reports show very few actual references to the CSC with the exception that a formal evaluation will be undertaken and reporting on the number of individual complaints received through the CSC.^{Footnote 3}  This makes it difficult to understand how in practice the CSC supports departmental strategies and outcomes. It is not part of the departmental performance story.

The CSC needs to tell its performance story in order to demonstrate its relevancy and contributions to FCAC strategic goals and to Canadian consumers. To better understand its business, the CSC will require clearly articulated strategic outcomes and performance metrics.

The CSC has the potential to develop in-depth expertise related to consumer issues that would be useful in informing the scope and approach of FCAC policies and assisting on complex issues. This in turn would support better alignment to Agency priorities and goals. Significant effort is made by the CSC to document individual complaints and to track referrals that are escalated but there appears to be a general lack of comprehensive analysis of those complaints to identify potential trends and common issues. Converting data into actionable intelligence has been limited.

There is a need for more strategic and formal engagement and provision of information to the Public Affairs Branch (PAB), Research, Policy and Education Branch (RPE) and the Supervision and Enforcement Branch (SEB). To a large extent, there is a lack of clarity on the type or level of information that should be provided although there was general agreement amongst internal stakeholders that the CSC data had the potential to be used as an ‘early warning system’ on emerging trends and issues to help manage risks and inform policy development. However, the CSC lacks the necessary data analytical capacity to provide analysis of its own data for the purpose of providing intelligence on emerging issues and trends to other business units/branches. This function would be the key point of contact with other business units to work on better integration of CSC performance and results with FCAC priorities and strategic goals. A CSC data analytics function would also support the development of more robust performance information including key performance indicators related to customer satisfaction, efficiency and economy. For its part, RPE, SEB and PAB should provide feedback on a more regular basis on whether and how CSC data is used if at all. There needs to be better collaboration and communication between the CSC and other business units with more formal protocols for data analysis and information sharing.

Significant effort is made to provide training and support to front line CSC staff who handle consumer enquiries and complaints. However, currently, there is no formal class-based training provided to CSC employees. Two types of training were identified: 1) external class-based formal training on call centre etiquette and dealing with difficult customers and 2) more regular internal briefings on new policies and initiatives and on core legislation and regulations (e.g. Bank Act).

To ensure better alignment of CSC services with Agency priorities and accuracy of information provided to consumers, it would also be helpful for the CSC to be aware earlier in the process of policies and plans. In general, early knowledge is useful to better respond to consumers and improves the level of trust with consumers.

Recommendations

Training

1. Provide regular class-based training on call centre etiquette for current and new employees in consultation with CSC employees leveraging existing training used by similar call or contact centres.
2. Revisit training manuals in consultation with CSC employees and the four contact centres that participated in the evaluation case study to determine whether and how they should be updated.
3. Similar to SEB and in consultation with CSC employees, establish regular monthly briefings between RPE, PAB and CSC to review core legislation (e.g. Bank Act) and regulation, key priorities and current challenges and risks.

Data Analytics

4. Establish a data analytics function in the CSC that would undertake strategic analyses of CSC data to identify emerging trends and issues and be the key point of contact to provide timely and relevant intelligence to internal FCAC stakeholders.
5. Establish a Data Management Roundtable comprised of data trustees and other key internal stakeholders to clarify roles and responsibilities for CSC data management, identify data sets and data sharing protocols and type and frequency of analyses of CSC information.
6. Develop an internal information and data sharing framework or plan based on the results of the Data Management Roundtable that could be implemented by the new CSC data analytics function.
7. Consider a facilitated workshop or retreat to articulate the CSC’s mandate and strategic outcomes.
8. Develop a Performance Information Profile (PIP) as a tool to articulate the CSC performance story including strategic outcomes and key performance indicators including customer satisfaction and service efficiency.
9. Report on one or more of the PIP key performance indicators in Agency business planning and reporting to demonstrate alignment with FCAC strategic goals and for public reporting purposes.

Providing Effective and Efficient Services

Interviewees collectively demonstrated a solid understanding of the core mandate of the CSC to support the Agency’s priorities and goals. The interviews also collectively demonstrate the perception that the CSC is performing well and is effective in terms of accessibility and timeliness in answering calls and responding to written correspondence. Indeed, it is consistently meeting its service standard of responding to 80% of written correspondence within 8 days. There are currently no service standards for telephone calls given the current technology used during Covid but the CSC is well known to respond to calls quickly and without prolonged wait times as can be the case with other call centres. Furthermore, there are plans to institute telephone call service standards and include an after-call survey function in the new Content Management System (CMS) system to measure agent performance.

Service efficiency needs to be addressed. There are currently no service efficiency metrics in place to assess whether the CSC is being delivered in a cost-effective manner. As efficiency metrics, Cost Per Call, Cost per FTE and Volume of Calls per FTE have not been collected prior to this evaluation, this information should be considered a baseline for future performance reporting and evaluations. This could be part of a data analytics function. Consideration should be given to ongoing liaison with the other Canadian government contact centres to compare such information where possible.

CSC has completed the first of two phases in the implementation of Microsoft Dynamics 365.^{Footnote 4}  The purpose of this project is to configure Microsoft Dynamics 365 to support business operations within the Consumer Services Centre (CSC). This includes configuring the Dynamics 365 system to handle CSC business processes that were supported by the Agency’s legacy Case Management System (CMS) WebCIMS, integration of Dynamics 365 with the Azure Voice Services telephony system (OmniChannel), and delivery of dashboard and reporting functionality required by the CSC.^{Footnote 5}  The expected result is improvement of CSC business workflows, reporting and analytics. To be successful, the data terms and categories used by RPE and SEB to input and extract data should be the same as the CSC where there are common data sets. It is not evident from the Data and Analytics Strategy that this will be the case. This should be a consideration when RPE and SEB develop their business glossaries to ensure data is collected consistently across branches. While there is some commonality in how data is entered in the CMS, further work is required to identify who owns what data sets and who is responsible for data quality and reporting on the data.

The CSC currently has the responsibility for Commissioner correspondence. This function was devolved to the CSC without an assessment of its impact on current resources and tasks. There is no separate reporting or statistics on the volume of correspondence per FTE. At the same time, interviews with some CSC staff revealed a resource pressure created by this task that affects other projects which include the development of training materials.

Feedback from the consumer survey indicated a total of 55% of consumers who contacted the CSC in the last few months disagreed or strongly disagreed that the CSC service met their expectations. A total of 57% disagreed or strongly disagreed that the information provided by the CSC assisted them in resolving their issue. A total of 35% disagreed or strongly disagreed that they received accurate information; 38% were neutral in their responses. Finally, 53% rated the overall quality of service provided as poor or very poor. A total of 48% rates the quality of service provided as excellent, good or average.

Discussions with some CSC staff suggest there is a general concern that it can be difficult to explain to customers that individual complaints cannot be resolved, compensation is not provided and managing expectations in this regard is a challenge. From an evaluation perspective, there is a risk that consumer expectations are not met and may result in a low satisfaction score. This may be caused by a lack of clear understanding of the FCAC mandate or lack of clarity in external communications. Therefore, it is important to note this context when assessing the survey results. For these reasons, the survey results should be considered as a baseline; effort is required to improve customer understanding of the scope and mandate of the CSC service.

Recommendations

10. Both CSC and FCAC external communications should clearly articulate the scope and mandate of the contact centre and its limitations.
11. The client satisfaction survey should be repeated, and the current data should be used as a baseline to assess improvements.
12. Develop and implement service efficiency performance indicators as part of the Performance Information Profile (PIP) including Cost Per FTE, Cost Per Call and Volume of Calls per FTE that will serve as a baseline for future performance measurement and reporting purposes. Consideration could also be given to First Call Resolution, Average Call Abandonment Rate and Transfer Rate.
13. As part of the implementation of the FCAC Data and Analytics Strategy, ensure that where appropriate there is a common vocabulary and data sets in CMS that can be shared between CSC, RPE, SEB and PAB.
14. Undertake an efficiency review of the volume of Commissioner correspondence to assess its impact on CSC deliverables and develop options for workload management.

Authority

This evaluation was included in the FCAC Business Plan 2022-23. The Plan commitment is to “complete a formal evaluation of the Agency’s Consumer Services Centre (CSC). The CSC performs a vital role in support of the Agency’s protection mandate by providing helpful information directly to financial consumers, merchants and stakeholders. The Agency will undertake an evaluation of the CSC to assess its current role and function, and to identify opportunities to maximize its capability to support consumers and contribute to FCAC’s strategic goals.”

Background

As the authoritative voice in matters of financial consumer protection, FCAC plays a critical role in the oversight of the federal financial sector in Canada. Consumer protection is important for Canadians. It builds trust and confidence in Canada’s financial system. FCAC achieves its mandate in 2 primary ways: by supervising federally regulated financial institutions and other regulated entities, and by enhancing the financial literacy of Canadians through research, education, and contributions to policy development. The Agency’s budgetary resources, employees and activities were organized around these two core responsibilities. In 2022–2023, FCAC transitioned to one core responsibility (see Departmental Results Framework) —the protection of financial consumers— to better reflect its integrated approach to fulfilling its mandate.^{Footnote 6}  It has two enabling programs – Supervision and Enforcement and Research, Policy and Education.

In 2018, the Government of Canada adopted legislation to enhance consumer protection and give FCAC additional powers by enhancing its supervisory authority. The legislation amended the Financial Consumer Agency of Canada Act and introduced a new Financial Consumer Protection Framework (The Framework) in the Bank Act. The Framework is an important milestone for consumer protection in Canada. To support its implementation, the government published the Financial Consumer Protection Framework Regulations in August 2021. The final elements of the Framework came into force on June 30, 2022.

In July 2021, FCAC released Make Change that Counts: National Financial Literacy Strategy 2021–2026. This renewed National Financial Literacy Strategy presents a vision of a Canada where everyone can build financial resilience in an increasingly digital world. FCAC consulted more than 90 organizations across the country to develop the strategy that builds on what was learned following the launch of the first strategy in 2015. The strategy also reflects the economic impact of the COVID-19 pandemic, considers the evolving financial marketplace, and includes new research insights. It focuses on actions stakeholders can take to improve the financial ecosystem and help Canadians build financial resilience. FCAC held targeted discussions with stakeholders to develop a measurement plan to assess the effectiveness of the initiatives set out in the National Financial Literacy Strategy. The plan was published in 2022.^{Footnote 7} 

For the Agency to deliver on its priorities, it requires efficient and timely support from all its branches and services. The overall objective of the evaluation of the Consumer Services Centre is to provide valuable information to assess the extent to which the service can support the Agency’s mandate and priorities as well as providing excellent service to Canadian consumers.

The FCAC has multiple and complex stewardship and compliance obligations related to legislative requirements including but not limited to the FCAC Act and the Financial Administration Act (FAA). This evaluation is not assessing Agency or CSC obligations with respect to legislative or regulatory requirements related to the FCAC.

Consumer Services Centre

Established in 2011, the Consumer Services Centre (CSC) interacts in a variety of ways with financial consumers and stakeholders to help carry out the Agency’s financial literacy and consumer protection mandates. The CSC does not resolve individual disputes and does not provide redress or compensation.^{Footnote 8}  

Some of the CSC’s key responsibilities include handling, triaging and documenting consumer enquiries. The CSC also reports on consumer complaints, provides consumers with information about complaints-handling, and directs them to financial information, resources and tools as needed. In its exchanges with consumers, and other stakeholders, the CSC sometimes handles sensitive and high-profile correspondence.

The CSC also collects and shares data internally with the intent of informing policy development and decision-making. Complaints received directly by the CSC are triaged to identify a potential breach relating to the regulated entities’ compliance with market conduct obligations. These complaints are then forwarded to the Supervision and Enforcement branch for further examination, when appropriate. For enquiries or complaints that do not relate to market conduct obligations overseen by FCAC, the Agency refers complainants to the appropriate regulatory body or consumer protection organization, where possible. In 2021–2022, FCAC received 4,745 consumer complaints. Of those, 295 were related to consumer protection measures that FCAC oversees and which were referred to the Supervision team for review.^{Footnote 9} 

Performance Measurement and Results

There has been steady and increasing emphasis on performance and results-based reporting for federal departments and agencies.

In July 2016, the Treasury Board of Canada (TB) issued the new Policy on Results. Performance measurement is defined by the Treasury Board Secretariat (TBS) as the process and systems of selection, development, and ongoing use of performance measures to guide decision-making.^{Footnote 10}  Performance measurement is a results-based management tool that federal organizations are required to use for decision-making, performance improvements and reporting. Performance measurement helps clarify expectations for service delivery, drive improvements and contribute to results-based management. It also reinforces accountability by making performance transparent.

The Corporate Services Branch has functional oversight for the establishment of Agency performance measurement and data collection and is responsible for the development of performance measurement for the Agency including the Departmental Results Framework (DRF) and Performance Information Profile or PIPs. Although the performance of Corporate Services will not be assessed as part of the Study, the Branch was used in an advisory capacity. It should be noted that as the FCAC is not funded through appropriations, it is not required to prepare a Departmental Plan or Departmental Results Report. Its annual reporting requirements are mandated by the FCAC Act and include a yearly business plan and annual report.

Service Excellence

There is also growing emphasis on fostering a culture of service excellence, ensuring services are citizen and client-centric and meet expectations for timeliness and quality. The TB Policy on Service and Digital outlines key service-related requirements for both external and enterprise-wide internal services.

The TB Policy on Service and Digital and supporting instruments serve as an integrated set of rules that articulate how Government of Canada organizations manage service delivery, information and data information technology, and cyber security in the digital era. The objective of this policy is to ensure “Client service experience and government operations are improved through digital transformation approaches” and service design and delivery is client-centric by design. The Policy on Service and Digital focuses on the client, ensuring proactive consideration at the design stage of key requirements of these functions in the development of operations and services.^{Footnote 11} 

Service is defined by the Policy as the “provision of a specific output, including information that addresses one or more needs of an intended recipient and contributes to the achievement of an outcome. Service standards are a public commitment to a measurable level of performance that clients can expect under normal circumstances. Service standards are an essential component in achieving service excellence and directly contribute to results-based management and achievement of results.^{Footnote 12} 

The TB Policy on Service and Digital applies to external and internal enterprise Government of Canada services. While departmental internal services are not included, the key principles are relevant. The Policy is intended to ensure that the design and delivery of services consider client needs (e. g. ease of access, accuracy, timeliness) and stewardship (policy compliance, security and integrity). The Policy requires that “service standards and real-time service delivery performance information for priority services are available to clients on Canada. ca for external services and on internal collaborative tools for internal enterprise services.” Other requirements include providing mechanisms for feedback from clients and addressing client service issues in a timely manner.

Under section 4.2.1.4 in the TB Policy on Service and Digital, Deputy heads are responsible for ensuring services have comprehensive and transparent client-centric standards, related targets, and performance information, for all service delivery channels in use, and this information is available on the department’s web presence.

Measuring and meeting service standards is essential for good client service and effectively managing performance. Service standards help clarify expectations for clients and employees and drive service improvement. Canadians gain confidence in the government when standards are met consistently. Departments are encouraged to allocate resources to meet any new improved service levels. Services that do not meet their standards risk eroding trust in government.

Collectively, these initiatives emphasize the growing importance of managing programs and services with a clear understanding of what they are trying to achieve, how success will be measured and at what cost. As such, it is expected that the CSC adhere to the same principles and requirements to which the government have committed to for external services, including ensuring that a robust performance measurement system is in place, client-centric service standards are developed and transparent reporting on results is provided to Canadians.

Evaluation objectives

The goal of the evaluation is to provide the Agency with strategically focused, objective, evidence-based information on the CSC. The objectives of the evaluation are to assess the CSC’s current role and function and identify opportunities to maximize its contributions.^{Footnote 13}  Specifically, the evaluation will assess the extent to which:

1. The CSC supports the priorities and strategic goals of the FCAC;
2. Effective and efficient services are being provided to financial consumers.

This evaluation assessed the program’s relevance and its performance in accordance with the Treasury Board Policy on Results. This is the first formal evaluation of the CSC. As such, the evaluation should be considered a baseline for subsequent evaluations and reviews of the service.

Methodology

The scope of the evaluation was established through interviews, a review of relevant TB policies (e. g. TB Policy on Results, TB Policy on Service and Digital) and related requirements. In addition, FCAC commitments related to performance measurement and service excellence including the Departmental Results Framework, Business Plan and Annual Reports were reviewed. Finally, research and analysis of relevant evaluations, studies and reports of similar contact centres were undertaken.

The period covered by the evaluation was from February to November 2023.

An evaluation matrix, including evaluation issues, questions, indicators and data sources, was developed during the planning phase (See Appendix B) with 6 lines of evidence including a CSC Client Satisfaction Survey.

Multiple methodologies were used to assess the program. These were:

• Data and document review: Data and documents were reviewed to provide information on the CSC and its context as part of the planning phase of the evaluation, and in determining the answers to the evaluation questions.
• Literature review: Literature was reviewed to obtain information on the CSC and its context as part of the planning phase of the evaluation, and in determining the answers to the evaluation questions including research of other jurisdictions to determine whether appropriate benchmarks exist.
• Interviews: Structured interviews were held with 22 key stakeholders from FCAC.
• Case study: A case study of 4 similar call centres was undertaken. The call centres are: Bank of Canada, Canada Deposit Insurance Corporation, Department of Finance and the Ontario Securities Commission. The scope and questions for the case study were derived from the contract’s Statement of Work.
• Survey: A customer satisfaction survey was undertaken.

It should be noted that the Final Report was completed and submitted on December 15, 2023 without a Management Response.

Findings

The following lines of enquiry and criteria are based on the contract Statement of Work, the TB Policy on Results, TB Policy on Service and Digital as well as standard evaluation questions regarding relevance, continued need and performance (effectiveness and efficiency). The findings and conclusions below are based on multiple lines of evidence that were used during the evaluation. They are presented by evaluation issue (relevance, continuing need and performance).

Relevance and continuing need

Line of Inquiry #1 (Internal): To what extent do CSC services support FCAC priorities and strategic goals? Is the CSC collecting useful data and information? What do the other branches do with the information received from the CSC? What additional value could the CSC provide to the branches at FCAC?

Alignment of CSC services with the FCAC priorities and strategic goals

In theory, the CSC aligns well with FCAC’s identified Departmental Results Framework (DRF) Strategic Outcome to protect financial consumers. The CSC also aligns well with the Business Plan’s strategic goals to “Be the authoritative source of Canadian financial consumer protection information” and “strengthen the financial literacy of Canadians for an increasingly digital world”. It should also be noted that the 2022-23 Business Plan called for the development of a marketing and communications strategy to ‘‘enhance awareness of its role as a regulator that protects financial consumers by supervising financial institutions and of its work with other federal regulators to uphold a financial system that consumers can have confidence in.”^{Footnote 14}  While the campaign did not take place, the Public Affairs branch pursues this objective through its ongoing communication and marketing activities. As such, there could be a consequent impact on the service, specifically an increase in the volume of enquiries. The CSC provision of services is also aligned with federal policies specifically the TB Policy on Service and Digital which is rooted in the principles of client centricity and a culture of service management excellence.

However, a review of corporate performance and planning documents including the Departmental Results Framework (DRF), Business Plans and Annual Reports show very few actual references to the CSC with the exception that a formal evaluation will be undertaken.^{Footnote 15}  This makes it difficult to understand how in practice the CSC supports departmental strategies and outcomes. It is not part of the departmental performance story.

Interviews revealed a consensus that in principle the CSC supports Agency priorities and goals generally but there is also a view that the CSC does not directly support operational needs in practice. For example, the current data inputted into the CMS system is largely not used or viewed as relevant. Interviewees indicated that, to be useful and relevant, data needed to be analyzed in aggregate format to identify trends and issues.

Potential for alignment of the CSC departmental strategic priorities as articulated in the Business Plan or DRF requires analysis. The following table is an illustration that suggests how the service could align with and support departmental priorities and strategic goals.

Potential linkages with CSC to Corporate Commitments

As the table demonstrates, there is potential to better incorporate the CSC into Agency planning and performance documents and reporting on performance. Doing so will require clearly articulated strategic outcomes and performance metrics for the CSC. This is addressed in greater detail in Line of Inquiry 3: Performance Measurement.

Internal stakeholder interviewees generally expressed a strong understanding of the importance of the service to ensure fair and transparent access to the Agency and support the FCAC mandate. In addition, the evaluation found a demonstrable interest in the service with respect to its potential as an advisory function to provide intelligence on emerging issues and trends. Interviews and documentary analysis indicate that the group has the potential to develop in-depth expertise related to consumer issues that would be useful in informing the scope and approach of FCAC policies and assisting on complex issues. This in turn would support better alignment to Agency priorities and goals.

Some interviewees considered that a possible future option would be to combine the CSC with other contact/call centres in particular those that fall under the Bank Act. Such a model may merit exploration to determine whether there is a business case from both effectiveness and efficiency perspectives including cost savings and better able to manage volume. The Canadian Deposit Insurance Corporation (CDIC) currently has an arrangement to provided telephone service by Service Canada in which CDIC is bundled with other organizations. This has resulted in some cost savings and service efficiencies and provides the ability to meet any unplanned volume increases in calls. However, as the call agents are not dealing exclusively with CDIC content, the knowledge learning curve is significant. Multiskilled agents cost more, need more training and are less efficient at each individual skill. But they provide greater flexibility in dealing with different types of services. In contrast, specialized agents cost less in the sense of wages and training and management can be easier. There is also higher turnover in multiskilled call agents which means continual reinvestment in training. This could lead to lower quality and accuracy of information being provided and increased reputational risks. To better manage these risks, CDIC is in the process of developing a Playbook for contact center services being provided by Service Canada. Better performance can be achieved by using a combination of specialized and multiskilled agents. Using a small number of multiskilled agents can make a difference by reducing the amount of training and providing greater flexibility in dealing with different types of services.^{Footnote 16}

The extent to which CSC information is used to inform FCAC decision-making and priority-setting.

The CSC currently provides regular reports on the volume of calls and written correspondence and the top three issues in bullet format. It also records individual contacts in the new Microsoft Dynamics Content Management System (CMS). It was noted in interviews by both CSC staff and representatives of the operational branches that there is a lack of engagement with other branches with respect to CSC information and data. It is typically one way communication with the CSC employee providing information on a potential issue to a particular business unit with little or no feedback on whether it was useful or used unless the branch (RPE or SEB) contacts them to require more information. It would be helpful for PAB, RPE and SEB to provide feedback on a more regular basis on whether and how CSC data is used if at all. It would also be helpful for CSC to be aware earlier in the process that guidelines/legislation are being worked on or a consultation is being planned. In general, early knowledge is useful to better respond to consumers and improves the level of trust with consumers. There is a need for the CSC to be seen as a continued internal extension of the SEB and RPE teams.

For its part, the CSC appears to be well integrated with other PAB business units and often uses the information developed by the consumer information team as a source of information for drafting and responding to consumers that contact the CSC.

From a data sharing perspective, there needs to be better collaboration and communication between the CSC and SEB and RPE business units with more formal protocols. In addition, greater clarity is required on the business process flow of information and data from the CSC to other Agency business units. It is unclear whether and when data inputted into the CMS is used by other business units and if so why. As a result, there are limitations to the CSC’s ability to maximize its value-added potential and contribute to decision-making and priority-setting. If the CSC better understood how the data was used or not used, it could lead to better data development and analysis that is more useful for SEB and RPE.

Significant effort is made by the CSC to document individual complaints and to track referrals that are escalated but there appears to be a general lack of analysis of those complaints to identify potential trends and common issues. In other words, the content of consumer enquiries appears to be well documented, but they are not sufficiently analysed to help uncover trends and emerging issues. Converting data into actionable intelligence has been limited. The CSC provides some actionable business intelligence, mostly to SEB but also to the Consumer Information team for webpages but it is not done regularly.

There are procedures in place for entering information in WebCims, the CMS system that had been used prior to the development of the new Microsoft Dynamics system daily task requirements, voicemails, escalation of complaints etc. These procedures are individually documented in emails and there does not appear to be a single centralized procedures document. In addition, there appears to be no formal criteria (e.g. level of risk) or process (e.g. Briefing notes) to relay issues for escalation. It would appear that issues are largely relayed upward via email with little or no accompanying analysis. This may explain in part why some internal stakeholders feel they do not have a good grasp of what the CSC does or its potential value-added role to support decision-making and priority setting by providing issues and trends analysis of its data. Interviews with key internal stakeholders validated this observation. CSC employees indicated that escalation or referral of complaints appears to be frequently a judgement call. As a result, while the ‘how’ is clearly defined, it is often unclear to CSC employees whether and when issues should be escalated.

The interviews revealed that roles and responsibilities for the analysis of CSC data were not well understood. Some CSC staff felt that they were not responsible for the data once it had been entered into the system; other branches felt that they were not the trustees of the data. Their expectation is that the CSC itself should be providing an analysis of the data. For its part, the CSC does not have the resources or expertise to do so.

The CSC should be a two-way facing entity ideally – external customers are Canadian financial consumers. But internal customers also exist - the two operational branches and in addition, PAB and Corporate Services. Internally, the CSC needs to be better positioned to transmit the right information at the right time to the right internal business unit or customer in order to support and inform decision-making and priority setting. For their part, PAB, SEB and RPE need to provide information on planned activities more proactively and earlier.

The extent to which FCAC operational branches are satisfied with the level of information provided.

The current approach is transactional and there is a need for more strategic and formal engagement and provision of information to PAB, RPE and SEB. Internal interviewees generally expressed the view that the current data was not useful or relevant for decision-making or priority setting; in most cases this was a result of a lack of trends analysis. Currently, the CSC data being stored in the CMS system are records of individual calls and written correspondence. The interviews at all FCAC levels revealed that to a large extent, there is a lack of clarity on the type or level of information that should be provided although there was general agreement amongst internal stakeholders that the CSC data had the potential to be used as an ‘early warning system’ on emerging trends and issues to help manage risks.

CSC has the requisite resources and tools to effectively support FCAC mandate and priorities.

The document review and interviews demonstrate significant effort to provide training and support to front line CSC staff who handle customer enquiries and complaints. However, currently, there is no formal class-based training provided to CSC employees. CSC employees feel that most training is done ‘on the job’ and they learn the type of information to be provided to customers primarily from pre-authorized “macros” and by checking with each other. However, it should be noted that there is a Reference Guide that contains examples of how to respond to specific types of complaints. There is also a CSC Assistant Handbook that details how to manage written correspondence. These documents are in the process of being updated. Therefore, it may be appropriate to revisit these guides with the CSC agents and the four call centres that participated in the evaluation case study to determine whether and how they should be updated. The Department of Finance has a training procedures manual “Guide to Responding to Public Inquiries in a Hybrid Workplace” (February 2023) that may serve as a best practice in this regard.

Training not only improves customer service, it can also help reduce employee turnover by improving employee satisfaction. CSC employees expressed the desire for more formal training in a number of ways to better support management of external consumers and internal provision of information and data. This is particularly important for new CSC employees. Two types of training were identified: 1) external class-based formal training on call center etiquette and dealing with difficult customers and 2) more regular internal briefings on new policies and initiatives and on core legislation and regulations (e.g. Bank Act). One model for training is the Ontario Security Commission (OSC) which leverages training on dealing with difficult customers through a local hospital. Similarly, the Bank of Canada leverages training from the Ottawa Distress Centre and Museum of History guide training. In addition, the OSC noted that keeping abreast of legislative, regulations and market trends is important; weekly meetings are held with the team to keep abreast of emerging issues. It was also noted in the interviews with CSC staff that the macros were out of date and needed to be refreshed.

Finally, as noted above there is no data analytics capacity in the CSC similar to that found in other branches (e.g. Radar). Such a function could be used for analysis of consumer complaints and other calls to identify emerging issues and trends. As the Data and Analytics Strategy states, as recently as fiscal year 2016-17, FCAC was a small agency with less than a hundred employees and a budget of less than $20M. The Report states: 

Building data and analytical capacity in the Agency is a key priority for the Strategy. The same needs to be true for the CSC; the CSC lacks the necessary data analytical capacity to provide analysis of its own data for the purpose of providing intelligence on emerging issues and trends to other business units/branches. Typically, the entity that collects the data is also responsible for analyzing the data to better understand its own business.

The afore-mentioned data analytics function would support the development of more robust performance information including key performance indicators related to customer satisfaction, efficiency, and economy. This function would be the key point of contact with other business units including Corporate Services, to work on better integration of CSC performance and results with FCAC priorities, strategic goals and related documents including the Departmental Results Framework (DRF), Business Plans and Annual Reports.

Recommendations

1. Provide regular class-based training on call centre etiquette for current and new employees in consultation with CSC employees leveraging existing training used by similar call or contact centres.
2. Similar to SEB and in consultation with CSC employees, establish regular monthly briefings between RPE, PAB and CSC to review core legislation (e.g. Bank Act) and regulation, key priorities and current challenges and risks.
3. Revisit training manuals in consultation with CSC agents and the four call centres that participated in the evaluation case study to determine whether and how they should be updated.
4. Establish a data analytics function in CSC that would undertake strategic analyses of CSC data to identify emerging trends and issues and be the key point of contact to provide timely and relevant intelligence to internal FCAC stakeholders.
5. Establish a Data Management Roundtable comprised of data trustees as identified under the Data and Analytics Strategy and other key internal stakeholders to clarify roles and responsibilities for CSC data management, identify data sets and data sharing protocols and type and frequency of analyses of CSC information.
6. Develop an internal information and data sharing framework or plan based on the results of the Data Management Roundtable that could be implemented by the new CSC data analytics function.

Line of Inquiry #2 (External): To what extent is there a continued need for CSC services? 

Legislative requirements

There is a continued need amongst Canadians for the FCAC Consumer Services Centre and the need to provide information from all channels (telephone, web, written correspondence).  The legislative authority for the CSC derives from the Bank Act (s. 627.54) which states: 

Volume of calls in comparison to other contact centres^{Footnote 18} 

The CSC had 7,767 total calls answered in 2022-23.  Total volume of all CSC contacts was 13,094 in 2022-23. This includes calls and written correspondence.  There was insufficient evidence in the case study with respect to total volume of contacts and total volume of calls of the four contact centres to undertake a comparative analysis.  Therefore, the CSC numbers should be treated as a benchmark for future performance analysis and reporting.

Performance

Performance assesses the extent to which the program has achieved its objectives and the degree to which it is able to do so in a cost-effective manner that demonstrates economy and efficiency.

Line of Inquiry #3: CSC has implemented performance measurement processes and practices that are sufficiently robust.

Current service standards and other performance indicators are appropriate and relevant:

A key purpose of including this line of inquiry is to assess and make recommendations on the current state of the CSC performance measurement and how the CSC can build a robust and relevant performance measurement system. The CSC performance measurement system requires strong leadership to mature the development and use of performance information to articulate results. There is evidence of effective leadership and good governance to support performance measurement. In addition, there is evidence that the current Agency governance structure supports the potential for integrated approaches into existing and planned performance and measurement and planning with respect to the CSC. However, in practice the CSC is not included in the Departmental Results Framework and there are few references to the service in Business Plans and Annual Reports. The CSC does report on certain metrics as part of the GC Service Inventory. However, to strengthen the CSC’s relevancy, it needs to tell a better performance story and be better integrated into current Agency planning and reporting processes. Furthermore, to develop a more robust performance measurement system, the CSC needs to articulate its mandate, mission and expected results. It may be worthwhile to consider a facilitated workshop or retreat.

A performance measurement framework or Program Information Profile (PIP) for the CSC may be useful. A PIP articulates the performance story, develops KPIs and service standards, drives data collection efforts and supports more strategic reporting to senior management. A PIP would also serve to link/align the CSC with the Departmental Results Framework. A CSC PIP can be a working tool and does not have to be formally part of the DRF. As an alternative to a PIP, insertion of 1-2 key performance indicators (KPI) in the Departmental Results Framework and any PIPs developed for RPE and SEB would be helpful to strengthen linkages between the service and departmental priorities and support more integrated data sharing, data management and reporting. This would also provide the necessary linkages to articulate how the CSC supports departmental priorities and goals and inform CSC’s contributions to the Business Plan.

A review of documentation provided reveals a strong focus on volumetrics – that is the number and frequency of calls and other forms of contact with the CSC. There is a lack of qualitative performance information with respect to CSC services including client satisfaction. This makes it difficult to assess service performance from a client-centric perspective. The evaluation found that there are two (2) service standards in place for written correspondence. Neither were designed or delivered considering client feedback as the CSC does not currently have a client feedback mechanism such as a client satisfaction survey.

There are currently no service standards in place for telephone calls due to the lack of technology although there was previously a call evaluation system prior to Covid. The new telephony system Omnichannel that was implemented in fall 2023 anticipates the use of the following criteria to assess call agent performance:

• Introduction
• Soft Skills (good grammar, demonstrates empathy etc)
• Information Navigation (able to find correct information)
• Product Knowledge (provide detailed and accurate information)
• Call Completion^{Footnote 19} 

These criteria taken together could be considered to be a Call Quality agent metric. Most facilities calculate this metric by giving each interaction an overall grade and it often includes multiple basic customer metrics such as those noted above. The new system also plans for an after-call survey that will measure client satisfaction. Call Duration is different than the Average Handling Time which is the average amount of time an agent spends on each call including the moment the call is picked up to when one of the parties hangs up and includes talk time, hold time and after call work time.

The CSC currently has the following service standards:

It should be noted that currently call recording is not possible with existing technology but is planned with the new Omnichannel telephony system.

Recommendations

7. Consider a facilitated workshop or retreat to articulate the CSC’s mandate and strategic outcomes.
8. Develop a Performance Information Profile as a tool to articulate the CSC performance story including strategic outcomes and key performance indicators including customer satisfaction and service efficiency.
9. Include reporting on one or more of the CSC PIP key performance indicators in Agency business planning and reporting to demonstrate alignment with FCAC strategic goals and for public reporting purposes.

Line of Inquiry #4: To what extent has the CSC achieved its expected outputs/outcomes?

Level of understanding of CSC’s current role and activities within the Agency

As noted earlier, interviewees collectively demonstrated a solid understanding of the core mandate of the CSC to support the Agency’s priorities and goals. However, its internal role to support the operational branches is less clear to stakeholders notably with respect to the data and information that should be provided to inform decision-making and priority-setting.

Level of client satisfaction in comparison to similar contact centres

The interviews collectively demonstrate the perception that the CSC is performing well in terms of accessibility and timeliness in terms of answering calls and responding to written correspondence. Indeed, it is consistently meeting its service standard of responding to 80% of written correspondence within 8 days. There are currently no service standards for telephone calls given the current technology used during Covid but the CSC is well known to respond to calls quickly and without prolonged wait times as can be the case with other call centres.

Accuracy of information is less well known as there are no performance metrics. However, the business process for learning on the job can be labour intensive. Agents have to develop their subject matter expertise from a set of some 600 macros or pre-established responses to enquiries. Updates to this information are not always done on a timely basis and there could be greater collaboration between operational branches (SEB and PAB) and the CSC in this regard.

Given that there are currently no performance KPIs, or performance information related to customer satisfaction, the evaluation pilot tested a customer survey. A traditional contact center metric, the customer satisfaction (CSAT) score measures customer satisfaction using survey data. Centres typically collect data for this metric by asking customers to rate their satisfaction level from 1 to 5. Accordingly, this was the methodology used to assess customer satisfaction with respect to CSC calls.

The following criteria were used:

• Service met expectations
• Resolution of the problem
• Accurate information
• Satisfaction with the overall quality of the service provided

The criteria are derived from the Institute for Citizen-Centred Service (ICCS) annual survey of call centers.

The consumer survey was posted on Canada.ca and the public was asked to fill it out if they had recently used the services of the CSC. Following their calls with consumers, CSC representatives would ask the consumer if they wanted to complete a survey about the service they had received. The survey was then emailed to consumers, or they were invited to complete the survey on the Canada.ca webpage when they identified privacy concerns with sharing email addresses. The survey was publicly posted and was available for anyone to complete it. For consumers who received a response from the CSC, they were sent the survey by email. The data used below is a combination of both the survey on Canada.ca and surveys emailed to consumers. The same survey was used for both. A total of 128 responses were received.

Question 1: How much do you agree or disagree that FCAC’s Consumer Service Centre met your expectations? 128 responses)

Question 2: How much do you agree or disagree that the information provided by FCAC’s Consumer Services Centre assisted you in resolving your issue? (128 responses)

Question 3: How much do you agree or disagree that you received accurate information? (128 responses)

Question 4: How would you rate the overall quality of the service provided? (128 responses)

    A total of 55% disagreed or strongly disagreed that the CSC service met their expectations.   A total of 57% disagreed or strongly disagreed that the information provided by the CSC assisted them in resolving their issue.  A total of 35% disagreed or strongly disagreed that they received accurate information; 38% were neutral in their responses. Finally, 53% rated the overall quality of service provided as poor or very poor.   A total of 48% rates the quality of service provided as excellent, good or average.

Discussions with some CSC staff suggest there is a general concern that it can be difficult to explain to customers that individual complaints cannot be resolved, compensation is not provided and managing expectations in this regard is a challenge.   From an evaluation perspective, there is a risk that consumer expectations are not met and may result in a low satisfaction score. This may be caused by a lack of clear understanding of the FCAC mandate or lack of clarity in external communications. Therefore, it is important to note this context when assessing the survey results.  For these reasons, the survey results should be considered as a baseline; effort is required to improve customer understanding of the scope and mandate of the CSC service.

Not surprisingly, the CSC client satisfaction survey results reflect a low level of understanding of the type of services that the CSC can provide.  The CSC is not a typical call or contact centre that resolves individual complaints.  It is constrained by its mandate and provides general knowledge and information.  It also acts as an intake for complaints which are referred to the SEB team for review.  CSC contact centre agents need to be equipped with clear and concise language to proactively explain their role and its limitations. In addition, general FCAC external communications (e.g. Webpages) should clearly articulate the purpose and scope of the CSC mandate and provide an explanation for its limitations as the FCAC contact centre.   This should be part of the overall effort to tell the CSC performance story. Once this is accomplished, the survey should be repeated, ideally within a two-year period.

Recommendations

10.  Both CSC and FCAC external communications should clearly articulate the scope and mandate of the contact centre and its limitations.
11. The client satisfaction survey should be repeated and the current data should be used as a baseline to assess improvements.

Line of Inquiry #5: To what extent has CSC been administered efficiently and economically?  Are the CSC human resources sufficient to fulfil its role? Are there other technological/ functionalities that the CSC could use to increase efficiency and effectiveness?

Demonstration of economy and efficiency is defined as an assessment of resource utilization (ie. program inputs) in relation to the production of outputs.^{Footnote 22}  Economy refers to minimizing the use of program inputs. Efficiency refers to the rate at which resources are used in the production of an output, with greater efficiency realized when the same level or a lower level of resources are used to produce a given level of output. A program has high economy and efficiency when financial resources inputs are minimized while outputs are maximized.

Use of technology and whether it is being optimized

FCAC is at a pivotal moment: an expanded mandate, a new legislative framework, the need to modernize business processes and the necessary supporting information technology renewal activities, are all in motion to overhaul and modernize the agency’s digital environment.  This is having an impact on the CSC.   The Consumer Services Centre has a mandate to receive direct complaints and enquiries from Canadian financial consumers relating to federally regulated financial entities and provides information to Canadians about financial products and services including their Banks complaint handling process. The prior CMS system and platform was at its end-of-life, severely outdated, and inadequate to support this mandate.

CSC has completed the first of two phases in the implementation of Microsoft Dynamics 365.^{Footnote 23}  The purpose of this project is to configure Microsoft Dynamics 365 to support business operations within the Consumer Services Centre (CSC). This includes configuring the Dynamics 365 system to handle CSC business processes that were supported by the Agency’s legacy Case Management System (CMS) WebCIMS, integration of Dynamics 365 with the Azure Voice Services telephony system (OmniChannel), and delivery of dashboard and reporting functionality required by the CSC.^{Footnote 24}  The expected result is improvement of CSC business workflows, reporting and analytics.  To be successful, the data terms and categories used by RPE and SEB to input and extract data should be the same as the CSC where there are common data sets.  It is not evident from the Data and Analytics Strategy that this will be the case.  This should be a consideration when RPE and SEB develop their business glossaries to ensure data is collected consistently across branches. While there is some commonality in how data is entered in the CMS, further work is required to identify who owns what data sets and who is responsible for data quality and reporting on the data.

Evidence of strategies to achieve outputs and outcomes economically and measures in place to assess efficiency and economy:

• Cost per Call
• Cost per FTE
• Total volume of enquiries/per FTE
• Total volume of Calls/per FTE

Perception of workload of CSC employees

Efficiency and economy need to be addressed.  There are currently no key performance indicators or benchmarks to measure CSC efficiency and economy.  As part of this evaluation, an analysis was undertaken of the cost per call and cost per FTE to illustrate the importance of these types of performance indicators.  Cost per call typically calculates the total costs both salary and O&M.^{Footnote 25}  The CSC cost per call is $68.11^{Footnote 26}  It should be noted that this is the first time this metric has been calculated and should therefore be used as a benchmark for future performance analysis and reporting.

Cost per FTE is also a useful metric. The following is not intended to be used as a comparison but is an example of a breakdown of some of the potential costs of an in-house team of 4 customer service representatives and a manager based on literature research:^{Footnote 27} 

• Salaries and Benefits: (USD) $141,284
• Customer Service Manager: $45,726
• Hiring costs: $20,645 
• Office space: $48,000
• Software and Hardware: $3,600

Other indirect operating costs not included in the above are training, IM/IT support, office supplies, fees, commissions plus other team members such as trainers, analysts, and quality assurance coaches. These may add up to an additional $200K USD per year. The total U.S. industry average total Cost per FTE for an in-house team of four (4) representatives and a manager is $459,255 USD per year, or $91,851 USD^{Footnote 28}  per employee or $125,680 CDN^{Footnote 29} . The Cost per FTE for the CSC is $156,885 CDN^{Footnote 30} .

Volume of Calls per FTE is another useful efficiency metric. CSC agents answered an average of 10.5 calls per day per person.^{Footnote 31}  Interviews revealed that the three CSC agents believed they were handling between 10-15 calls per day per person and felt that they were able to handle the current call volume and in general, the total number of contacts including calls, and written correspondence. CSC handled an average of 13,094 total contacts including calls and written correspondence from April 2022 to March 2023. This should be considered a benchmark for future performance analysis and reporting.

As efficiency metrics, Cost Per Call, Cost per FTE and Volume of Calls per FTE have not been collected prior to this evaluation, this information should be considered a baseline for future performance reporting and evaluations. This could be part of a data analytics function. Consideration should be given to ongoing liaison with other Canadian government contact centres to compare such information where possible.

The CSC may wish to also consider the following call metrics: First Call Resolution, Average Call Abandonment Rate and Transfer Rate. In particular, the Transfer Rate is how many interactions are sent to another department or agency. Keeping this number as low as possible generally leads to better results. The average U.S. industry transfer rate is 9.9% for call centres.^{Footnote 32}

Another metric is First Call Resolution (FCR) which means providing a resolution to the customer’s query in the first attempt and is considered a critical driver of customer satisfaction. According to reports, U.S. private sector call centres showing an 85% FCR score had lower operating costs and higher employee satisfaction.^{Footnote 33}  The global call center first contact resolution rate is between 70-75%. The benchmark suggests that most call center agents can resolve customer problems on the first try without following up or doing extra tasks.^{Footnote 34}  First Call Resolution rates also help identify which agents need further assistance. Customer satisfaction and First Contact Resolution were the two most popular metrics to measure success in call centres.^{Footnote 35}

It should be noted that the CSC currently has responsibility for Commissioner correspondence. This function was devolved to the CSC without an assessment of its impact on current resources and tasks. There is no separate reporting or statistics on the volume of correspondence per FTE. At the same time, interviews with some CSC staff revealed a resource pressure created by this task that affects other projects which include the development of training materials. Consideration could be given to an assessment of the volume and impact of Commissioner correspondence on the unit and whether it should be transferred back to the Commissioner Secretariat to free up resources necessary for the development and implementation of written training materials and address other CSC pressures. Further analysis of the level of effort required to address Commissioner correspondence is required.

Recommendations

12. Develop and implement service efficiency performance indicators as part of the Performance Information Profile (PIP) including Cost Per FTE, Cost Per Call and Volume of Calls per FTE. Consideration could also be given to First Call Resolution, Average Call Abandonment Rate and Transfer Rate.
13. As part of the implementation of the FCAC Data and Analytics Strategy, ensure that where appropriate there is a common vocabulary and data sets in CMS that can be shared between CSC, RPE, SEB and PAB.
14. Undertake an efficiency review of the volume of Commissioner correspondence to assess its impact on CSC deliverables and develop options for workload management.

Line of Inquiry #6: Case Study

The information gathered from the four case study participants – Bank of Canada, Canada Deposit Insurance Incorporation, Department of Finance and the Ontario Securities Commission - was highly anecdotal in nature and specific to each entity making it difficult to undertake a horizontal analysis or identify key themes or trends. Some best practices with respect to training have been included in this Report for consideration. In addition, the evaluation sought information on efficiency metrics to obtain a mean average that could usefully serve as a baseline for CSC comparison purposes but this information was largely unavailable.

Conclusion

The CSC needs to be better positioned to support FCAC strategic priorities and goals. There is a need for improved communications and information sharing between the CSC and the two operational branches – Research, Policy and Education (RPE) and Supervision and Enforcement (SEB) as well as Public Affairs (PAB). The CSC lacks the essential data analytics function to provide internal stakeholders with an analysis of emerging trends and issues to inform policy development and decision-making, establish written data sharing protocols and act as the key liaison with other branches. For its part, PAB, RPE and SEB need to strengthen its communications with the CSC by providing more timely briefings on Agency initiatives and clarify what types of CSC data are useful.^{Footnote 36}  A Data Management Roundtable could help to clarify roles and responsibilities for CSC data management, identify data sets and data sharing protocols, type and frequency of analyses and consider an information sharing framework or plan that could be implemented by a new CSC data analytics function.

The CSC could benefit from a more strategic alignment with current performance and planning exercises including strategic planning and annual reporting. Consideration should be given to the development of a Program Information Profile (PIP) as a tool to articulate the CSC’s expected results and key performance indicators that support both Agency and CSC business planning and performance reporting on customer satisfaction and efficiency. A CSC PIP does not have to be a formal part of the Departmental Results Framework (DRF).

About the Evaluation

Authority

The Commissioner of the FCAC approved this evaluation as part of the 2021-22 Business Plan.

Evaluation Objectives

This evaluation examined the Contact Services Centre (CSC). The program is administered by Public Affairs branch. The service is not identified in the FCAC Departmental Results Framework. However, it supports a “Fair, open and transparent acquisition that provides best value to Canadians and is delivered effectively and efficiently to the satisfaction of government and Canadians.”

The objectives of the evaluation were to assess the extent to which:

1. The CSC supports the priorities and strategic goals of the FCAC;
2. Effective and efficient services are being provided to financial consumers.

Approach of the Evaluation

The evaluation was conducted in accordance with the Standard on Evaluation for the Government of Canada. The evaluation took place between February and November 2023 and was conducted in four phases: pre-planning, planning, examination, and reporting. An Evaluation Matrix was developed using the Treasury Board’s five core issues to be addressed in evaluations. The following data sources were used to inform the lines of evidence to answer the questions in the Evaluation Matrix:

• Data and document review: The document review involved an assessment of primary documents that provided information on the program and its context for the planning phase and also provided information for the assessment of the evaluation questions. The primary documents included program administration, monitoring and reporting documents.
• Literature review: The literature review involved an assessment of secondary documents that provided information on the program and its context for the planning phase and also provided information for the assessment of the evaluation questions.
• Interviews: Twenty two structured interviews were held with key departmental officials.
• Case Study: A case study of 4 similar call centres was undertaken. The call centres are: Bank of Canada, Canada Deposit Insurance Incorporation, Department of Finance and the Ontario Securities Commission. The scope and questions for the case study were derived from the contract’s Statement of Work.

Limitations of the Evaluation

While the evaluation relied upon a broad range of information sources, both qualitative and quantitative, it is important to note that there are limitations to social science research. Key evaluation issues were targeted during the evaluation to respond to TB Policy on Results questions, whereas the CSC ’s overall performance could have been measured in a number of other ways. In addition, as the evaluation team was unable to directly compare the program’s performance against an identical program in another jurisdiction, and every jurisdiction bases their call centre service functions on different principles, it was not possible to benchmark the program against a similar program. Therefore the case studies were used to inform the evaluation lines of inquiry in a general way.

Reporting

Based on the lines of evidence, a Director’s Draft Observations and Conclusions was prepared for Validation. Subsequently, it was sent to the Assistant Commissioner, Public Affairs responsible for the CSC for review and comments. Validation of facts and any other comments on the Director’s Observations and Conclusions for Validation were assessed and, where appropriate, incorporated. The draft Final Report including recommendations was then sent to the Assistant Commissioner, Public Affairs to develop the Management Responses to the Recommendations.

Project team

The evaluation was conducted by an independent consultant supported by the Manager, Consumer Services Centre and the Director, Consumer Services & Information and under the overall direction of the Assistant Commissioner, Public Affairs. The evaluation was reviewed by the FCAC Head of Evaluation.

Annex A: Glossary of terms

Economy: minimizing the use of resources. Economy is achieved when the cost of resources used approximates the minimum amount of resources needed to achieve expected outcomes.

Efficiency: The extent to which resources are used such that a greater level of output/outcome is produced with the same level of input or, a lower level of input is used to produce the same level of output/outcome. The level of input and output/outcome could be increases or decreases in quantity, quality, or both (TB Policy on Results).

Effectiveness: The impacts of a program, policy or other entity, or the extent to which it is achieving its expected outcomes (TB Policy on Results).

Performance Measurement: The process and systems of selection, development and ongoing use of performance measures to guide decision-making (TBS Results-based Lexicon)

Relevance: The extent to which a program, policy or other entity addresses and is responsive to a demonstrable need. Relevance may also consider if a program, policy or other entity is a government priority or a federal responsibility.

Service: Provision of a specific output, including information that address one or more needs of an intended recipient and contributes to the achievement of an outcome (TB Policy on Service).

Service Standard: A public commitment to a measurable level of performance that clients can expect under normal circumstances (TBS Guidance on Service Standards).

Annex B: Evaluation matrix

Questions - relevance

Line of inquiry #1 (internal)

• To what extent do CSC services support FCAC priorities and strategic goals?
• Is the CSC collecting useful data and information?
• What do the other branches do with the information received from CSC?
• What additional value could the CSC provide to the branches at FCAC?

Indicators

• Perception that there is alignment of CSC services with the FCAC Repriorities and strategic goals
• Assessment of the linkages between CSC functions and FCAC mandate, priorities and strategic goals
• CSC information is used to inform FCAC decision-making and priority-setting.
• FCAC operational branches are satisfied with the level of information provided.
• CSC has the requisite resources and tools to effectively support FCAC mandate and priorities.

Data sources and methods

• Document review
• Literature search
• Interviews

Risks and risk mitigation strategies

• There is a risk that the role of CSC is not well understood or cannot be explained by key stakeholders. This risk will be mitigated by assessing the linkages between CSC functions and FCAC mandate and strategic goals.

Line fo inquiry #2 (External)

• To what extent is there a continued need for CSC services?

Indicators

• Volume of calls in comparison to similar call centre entities.
• Client satisfaction rate (CSAT) in comparison to other call centres
• Legislative requirements

Data sources and methods

• CSC data
• Case Study
• Literature Search
• Client Satisfaction Telephone Survey
• ICCS CMT Data

Risks and risk mitigation strategies

• There is a risk that the volume of calls and client satisfaction rate may not be comparable to other similar entities.  Risk will be mitigated by undertaking a case study and literature search as well as use of ICCS CMT.

Questions - Performance (Effectiveness, Efficiency, Economy)

Line of inquiry #3

CSC has implemented performance measurement processes and practices that are sufficiently robust

Indicators

• Current service standards and performance indicators are clearly defined, appropriate and relevant (see Evaluation Terms of Reference for details)

Data sources and methods

• TB Policy on Results
• TB Policy on Service and Digital
• Comparative Analysis of KPIs in comparison to ICCS CMT data
• Document Review
• Literature Search

Risks and mitigation strategies

• The current KPIs are largely focused quantitatively on volume and do not address qualitative value e.g. client satisfaction. Risks will be mitigated by
• Undertaking a client satisfaction telephone survey.
• Benchmarking via case study and ICCS CMT data.
• There is a risk that there is a lack of capacity to support effective performance measurement and improvement of the CSC. This risk will be mitigated by evidence-based recommendations in the final report to support the development of a robust performance measurement system.

Line of Inquiry #4 (Effectiveness):

To what extent has the CSC achieved its expected outputs/outcomes?

Indicators

• Level of understanding of CSC’s current role and activities within the Agency.
• Level of client satisfaction in comparison to similar call centres (See Evaluation Terms of Reference for details).

Data sources and methods

• Interviews
• Client Satisfaction Telephone Survey
• ICCS CMT data comparative analysis
• Literature search 

Risks and risk mitigation strategies

• There is a risk that consumer expectations are not met/cannot be met because they exceed what can be delivered within the limitations of the FCAC’s and CSC’s mandate.  This risk will be mitigated by undertaking a comparative analysis with similar call centres to benchmark effectiveness and efficiency and a comparative analysis with ICCS CMT data.
• There is a risk that CSC expected outputs and outcomes are not clearly defined as there is no logic model or PIP for the program.  This risk will be mitigated by undertaking a comparative analysis of client satisfaction with similar entities via a case study and the ICCS CMT data.

Line of Inquiry #5: (Efficiency and Economy):

• To what extent has CSC been administered efficiently and economically?
• Are the CSC human resources sufficient to fulfil its role?
• Are there other technologies/functionalities that the CSC could use to increase efficiency and effectiveness?

Indicators

• Clearly defined and standardized business processes
• Measures in place to assess efficiency and effectiveness.
• Total Call volume/FTE
• Cost per call/FTE
• Perception of workload of CSC employees
• Use of technology and whether it is being optimized (Case Study)
• How data and intelligence flows from the CSC to the operational branches (TBD)

Data sources and methods

• Interviews
• Document Review
• Literature Search
• Comparative analysis with ICCS CMT data if available.

Risks and risk mitigation strategies

• There is a risk that CSC expected outputs and outcomes are not clearly defined as there is no logic model or PIP for the program. This risk will be mitigated by undertaking a comparative analysis of efficiency and economy using ICCS CMT data.

Line of Inquiry #6: Case Study

• A case study of four other call centres with similar characteristics and mandate to the CSC will be undertaken.

Indicators

• What in your words is the purpose of your call centre? Please describe.
• Has a formal evaluation or review been undertaken? If so, would you be able to share it with FCAC?
• What are the key successes of your call centre? What have been the greatest challenges? What are the best practices that FCAC could leverage?
• What are your volumes (call and correspondence) and your employees’ workloads at your call centres?
• What levels, skillsets or competencies do teams in similar call centres have that could benefit the CSC (e.g. data analysts)?
• What technologies are your call centres using?
• What are the service standards of your call centres and how is performance measured?
• Who is your customer internally and externally?

Data sources and methods

• Case Study Interviews
• ICCS CMT data

Risks and risk mitigation strategies

• There is a risk that the call centres do not have relevant performance information on effectiveness, efficiency, and economy. The case study will rely upon their existing metrics and service targets. This risk may be mitigated by use of the ICCS CMT data to supplement findings and address data gaps.