Evaluation of the Human Resources Services Modernization Initiative

Implementation of CHRBP

The review of administrative data and documents confirmed that the CHRBP was implemented in the participating departments. The first step in the implementation of the CHRBP was for these departments to identify the gaps and opportunities associated with implementing the new processes. Participating departments identified 179 gaps and 538 opportunities across the seven CHRBP processes. By March 31, 2014, all departments had fully addressed these gaps and opportunities to Level 4. However, a few interviewees and evidence from one of two CHRBP case studies revealed that departments were encouraged to identify gaps and manageable opportunities in process areas where they felt they could implement the changes by the deadline. Thus, a number of potential opportunities were not pursued in favour of achieving the deadline. As well, according to some interviewees, common HR business processes would only be achieved when implemented at Level 5.

Finally, a few respondents said that they did not feel that all of the CHRBP has in fact been fully implemented. These respondents indicated that their departments continued to use processes previously in place.

Initial adoption of GC standard

As described above, the initial pilot of the GC standard involved a cluster of five departments implementing PeopleSoft v8.9 (Agriculture and Agri-Food Canada, the Canadian Food Inspection Agency, Fisheries and Oceans Canada, Health Canada and the Public Health Agency of Canada). According to all lines of evidence, the five departments successfully implemented PeopleSoft v8.9. The evidence from the document review and interviews suggests this initial adoption achieved its expected outcomes and encountered few challenges.

Under the HRSM initiative, four departments (Library and Archives Canada, Canadian Institutes of Health Research, Public Services and Procurement Canada, and Employment and Social Development Canada) were to have implemented PeopleSoft v9.1 by March 31, 2015. The evaluation found that while the basic system (PeopleSoft v9.1) was implemented, functionalities such as the reporting module would only become operational in the future under the My GCHR initiative.

Contribution to better alignment of HR systems investments to GC-wide priorities

All lines of evidence found that by moving to a single configuration of PeopleSoft v9.1 and to the CHRBP, the HRSM initiative has begun to align HR systems investments for those departments that participated in the HRSM initiative to modernize back-office systems. The My GCHR initiative, through the adoption of PeopleSoft v9.1 by additional departments, continues to reinforce the alignment of HR systems investments with GC-wide priorities.

According to key informants and case studies, although the HRBSP component sought to implement PeopleSoft v8.9 rather than v9.1, HRBSP nevertheless played an important role in aligning HR systems investments for participating departments and demonstrated that moving to a common HR system was possible.

Interoperability and contribution to one set of data

The document review and interviews with key informants confirmed that an interoperability framework^{Footnote 4} was developed under the HRSM initiative. Because a second phase of the interoperability component was not funded, the GC standard^{Footnote 5} for interoperability was not developed as planned. Subsequently, Employment and Social Development Canada used the framework to build a prototype.^{Footnote 6} Key informants indicated that the main impact of not proceeding with the component was a delay in the achievement of one intermediate outcome: “one set of reliable data and improved data exchange across the GC back office.”

Outside the context of the HRSM initiative, according to documents and key informants, those at the Secretariat responsible for interoperability worked with Employment and Social Development Canada to implement an architecture and solution platform^{Footnote 7} based on the interoperability framework. This platform is described in documentary evidence as “repeatable,” and it is anticipated that this work can be “lifted and shifted … to serve the entire Government of Canada.”^{Footnote 8}

With one system (PeopleSoft v9.1), the ability to have one reliable set of data is possible. However, without interoperability fully realized and the absence of a reporting module in the system as part of the HRSM initiative, one reliable set of data had not been achieved.

Contribution to more consistent efficient and effective HR service delivery across the GC

The reviews of documents and administrative data showed some progress toward more consistent HR service delivery across pilot departments because of the CHRBP and an increase in self-service functionality and automation of some routine HR transactions. Examples identified to evaluators included a new centralized HR structure that holds HR staff accountable, the completion of the outstanding signatures of directors general on organizational charts, documentation of grievances and employee awards in employee files, and standardized tools.

According to documents and comparisons between baseline and performance data, improvements have been recorded in the satisfaction of managers with the quality and timeliness of HR advice, in HR processing times, in service standards for staffing and classification, and in the assignment of competencies to work descriptions.

Furthermore, as evidenced in the Management Accountability Frameworks for the 20 largest departments, departments reported wide-ranging benefits from CHRBP implementation, including improvements in consistency, efficiency, effectiveness, technology, engagement, clarity of roles, service delivery and governance.

Nevertheless, concern was voiced by a few respondents about the sustainability of the HRSM components within departments, owing to lack of monitoring after implementation. The evaluation found evidence^{Footnote 10} among some interviewees that their departments had not fully implemented the new CHRBP standard or PeopleSoft v9.1 and, of these, a few have continued to operate parallel or “shadow” HR shops to offer direct support to managers or HR-related information systems. According to these key informants, shadow HR shops exist because not all departments buy into or understand the new allocation of accountabilities (for example, between HR professionals and managers) required under the CHRBP and because of issues with the new software either not generating reports or because of software incompatibility. These few departments indicated that they would continue to run parallel systems until PeopleSoft v9.1 is fully functional and their confidence in the new system increases.

Contribution to lower GC-wide total cost of HR management system ownership

It is too early in the HRSM initiative for evaluators to fully assess the achievement of the long-term outcome, given that departments are still adopting PeopleSoft v9.1^{Footnote 11} (My GCHR) and that the interoperability component was not developed. Consequently, estimates of lower GC-wide costs of HR management system ownership were not examined.

However, evaluators tried to identify instances where the initiative contributed to lower GC-wide costs of HR management system ownership in the document review, which identified some savings through cost avoidance. Although interviewees also confirmed avoidances, there is limited documentary evidence that costs have in fact dropped. The document review of project documentation noted that possible cost avoidances might occur under the HRBSP and PeopleSoft v9.1 Joint Build because departments would no longer undergo individual system upgrades and because the central and standard system would manage upgrades across the enterprise.

Key informants generally believed that the ownership costs of HR management systems are lower as only one licence is now required and there are fewer older systems requiring costly maintenance. However, there were no data on ownership costs to help evaluators determine the contribution to cost reduction. Notwithstanding the observations of the key informants described above and evidence of possible cost avoidance in documents, the use of parallel systems by a few departments would delay expected cost savings and, in some case, increase HR costs, particularly for departments operating two systems.

Improved data exchange, information, interoperability capability and capacity across the GC back offices

As discussed in the “Evolution” section, evaluators were unable to assess the achievement of this outcome because this HRSM activity ended after funding earmarked for the second year of the project was removed from the fiscal framework. As a result, the project was terminated in 2013. Since work on the interoperability standard has resumed within the Secretariat, this activity could be examined as part of a future evaluation of My GCHR.^{Footnote 12}

Efficiency

As noted earlier, actual costs savings were not determined through the evaluation since it is too early in the initiative’s life cycle to identify them. However, the evaluation attempted to learn which delivery approaches might contribute to decreased costs. These approaches included the Secretariat’s role in early planning and support to departments, the willingness of departments to invest in the process, and collaboration between departments and the Secretariat.

To improve the efficiency of the Secretariat’s management of government-wide transformation initiatives, the evaluation evidence identified alternatives and improvements. Earlier and better consideration of the context (for example, size and structure) and capacity of small departments would have resulted in the development and dissemination of tools and guidance for small departments that could have improved the efficiency of the implementation of the CHRBP standard as well as PeopleSoft v9.1 in pilot departments. Consideration of context would also have included recognition that participation in complex transformation initiatives represents a financial burden for small departments:

• efficiencies would have been gained if opportunities to learn from other departments had not been limited
• more, earlier and better involvement, communication, coordination and tools provided by the Secretariat would have facilitated the implementation by departments, particularly of the CHRBP standard, where some departments did not fully understand the standard or the expectations of their participation^{Footnote 13}
• less siloed GC-wide governance and business models and clearer mandates would have resulted in more efficient use of time at meetings, but also in more efficient use of time between meetings

Governance and delivery

Key informants’ views on the governance approach for the HRBSP component were positive overall, though the governance of the other components and the overall initiative were viewed as uneven and feedback was mixed.

The document review and interview evidence revealed that the Secretariat was at times unclear about how it set direction as the business owner. Moreover, evaluators observed that there was no central oversight to coordinate all HRSM components. Rather, each component was governed and operated independently. In the view of the evaluators, this led to fewer opportunities to adapt to the changes within components. For example, better coordination could have helped communicate the impacts of the termination of interoperability funding, and processes could have been standardized to a lower level of detail to facilitate system implementation.

For the PeopleSoft v9.1 Joint Build component, some interview respondents indicated that governance was ineffective. Some key informants and case study participants identified tensions among decision makers that necessitated the escalation of issues to higher levels of management. This caused delays and created operational risks in the delivery of the component. Regarding clarity of roles and responsibilities, there were mixed views, with some respondents indicating they were clear and a few stating that information on who was doing what was confusing. The department’s team primarily dealt with Public Services and Procurement Canada, and the Secretariat was never involved in the decision‑making process. The governance structure for the PeopleSoft v9.1 Joint Build had to be adapted along the way in order to clarify roles and responsibilities.

Regarding the interoperability component, interviewees familiar with it felt that the approach to governance was siloed and that there was poor understanding of the component, resulting in ineffective project oversight.

CHRBP interdepartmental engagement received more positive feedback, particularly for the clarity of roles and responsibilities and structure. A few respondents felt it was ineffective in terms of decision making and influence. In terms of central governance, some respondents felt that the CHRBP’s centrally led governance structure was not relevant for them because their departments did not participate in it. A few respondents explained that the central governance did not address their departments’ needs for support in the implementation of the standard.

Lessons learned

Beyond the efficiency-related findings, evaluators identified the following additional evidence:

• Large-scale GC-wide transformations should establish a governance approach whereby consensus is encouraged and delays are avoided by having a single authority for decisions.
• Governance and other project characteristics of GC‑wide transformation initiatives should recognize cross-functional roles and how the initiative plays a role in, and is influenced by, larger initiatives.
• Interdepartmental collaboration and partnerships, sharing and support should be encouraged early and should be supported by the Secretariat through the sharing of tools, including IT platforms, and meetings.
• The timing and degree of implementation of the initiative should consider a department’s capacity to implement it. Where capacity is limited and additional funding or support is not available, allowances should be made early in the process to manage expectations.
• GC-wide transformation initiatives should be monitored after implementation to determine whether outcomes are achieved, to identify where interventions are required to achieve outcomes, and to measure cost savings and other efficiencies. Post-implementation monitoring would also reinforce the importance of the initiative among departments, and if the monitoring reveals cost savings have been achieved, departments will be more likely to buy into subsequent GC-wide transformation initiatives.

Relevance

1. Does the HRSM initiative address a continuing need?
2. To what extent are the objectives of the HRSM initiative aligned with government priorities and the Secretariat’s departmental strategic outcome?
3. To what extent is the HRSM initiative aligned with federal roles and responsibilities for HRSM within the federal government?

Performance (effectiveness, efficiency and economy)

4. To what extent has the HRSM central governance contributed to more coordinated implementation and transformation of the HR service delivery across the GC?
5. To what extent have departments^{Footnote 14} implemented the Common HR Business Process?
6. To what extent has the initial adoption of the GC standard (PeopleSoft) occurred?
7. To what extent has the HRSM initiative contributed to better alignment of HR systems investments to GC‑wide priorities?
8. To what extent has the HRSM initiative contributed to one reliable set of data?
9. To what extent has the HRSM initiative contributed to consistent, more efficient and effective HR service delivery across the GC?
10. To what extent has the HRSM initiative contributed to lower GC-wide total costs of HR management system ownership?
11. What progress was made towards the development of the GC interoperability framework? What are the implications to the achievement of outcomes from the GC interoperability component not proceeding as planned?
12. Are there more effective solutions to meeting the needs addressed by the HRSM initiative?
13. What factors have contributed to the achievement of the expected outcomes of the HRSM initiative? What, if any, have been the main barriers to achieving these outcomes?
14. Was the delivery of the HRSM initiative efficient?

Document and data review

The review of documents and administrative data provided evidence for most evaluation questions. Documents were an important source of information to answer evaluation questions related to relevance and performance. Key documents and data for the review included project charters, close-out reports, Management Accountability Framework (MAF) reports, progress reports and monthly departmental reports, reports to Treasury Board ministers, Employment and Social Development Canada’s implementation schedule and progress reporting, HRIS progress reports and monthly reporting (dashboard), project status reports, lessons learned documents, HRSM governance decisions, meeting minutes, project documentation, onboarding schedules, lessons learned documents, close-out and evaluation reports, and program and baseline performance data. The review was conducted using a review grid to gather the information for each source and each evaluation question. The draft template was pre-tested by the primary reviewer, and all documents were catalogued.

For the administrative data review, in order to incorporate administrative information into this report, all baseline and performance measurement data (submitted in the form of individual Excel databases) were collated. The end result was one database with baseline submission data and performance measurement data side by side for each of the 20 largest departments. Further, an analysis of an online Access database (CHRBP tracking tool) was completed to track gaps and opportunities identified by departments. From this consolidated database key findings were drawn to help inform the evaluation questions.

Key informant interviews

Key informant interviews are a qualitative method used in evaluation. They gather views and factual information from key informants selected from within the federal government and the core public administration. For the evaluation of the Human Resources Services Modernization (HRSM) initiative, the key informant interviews were used to address all 14 evaluation questions in the evaluation framework.

Many different sources were approached to participate in the key informant interviews, including those familiar with the HRSM initiative overall and those familiar with only one or two components of the initiative. From an overarching perspective, members of the Human Resources (HR) Services Modernization Advisory Committee were consulted, as well as members of the HR Services Modernization Secretariat of the Office of the Chief Human Resources Officer (OCHRO). As well, a sample of representatives from various departments that participated in one or more HRSM initiative components was consulted. Where appropriate, some Secretariat and Public Services and Procurement Canada (PSPC) respondents were consulted about their department’s implementation of the HRSM components.

For the Common Human Resources Business Process (CHRBP) component, a sample of the 84 departments that adopted the CHRBP was developed with input from OCHRO. Selection criteria for departments included knowledge of the CHRBP activities, outputs and outcomes at their department, and whether the department had a positive or challenging experience. Selection of departments using these criteria was based on recommendations from OCHRO. Respondents were stratified by size of department (large, medium and small).

Of the 55 interviews planned, 51 interviews were conducted. The number in the “Departments” column (Table 1) is not necessarily the number of departments where interviews were conducted but rather the number of individual interviews conducted with departmental representatives, as in some cases more than one interview was conducted for some departments. Some respondents were able to speak to the evaluation questions from multiple perspectives (for example, many of the Secretariat’s representatives could speak from an overall perspective as well as about a specific component, and many departments were involved in both the CHRBP and the PeopleSoft v9.1 Joint Build project or the Human Resources Information System (HRIS)). The numbers in Table 1 indicate the component for which respondents were specifically identified, except for the Secretariat’s respondents, who had a broader perspective than expected, in which case they were included in the “overall” respondent type. However, when respondents provided evidence for other components, the analysis took their views into account.

Departments were invited to organize group interviews if they preferred. However, each interview of this type would still be weighted as one interview regardless of the number of departments and the number of interviewees in each interview group. The numbers in Table 1 are the number of interviews and not the number of interviewees.

The actual number of respondents who shared a particular opinion is not presented in this report. This is because in most cases, there was a small total number on which to base the findings, and thus the presentation of a number of respondents would be misleading. In many cases, owing to variations in the respondents’ familiarity with the subject matter and their resultant ability to answer some of the question, the total number providing evidence for an evaluation question could be quite low; sometimes only two or three people shared an opinion.

Instead, a qualitative scale was used to depict the proportion of those who answered a question and who shared an opinion:

• Large majority: findings reflect the views and opinions of at least 75% of respondents
• Most: findings reflect the views and opinions of at least 50% but less than 75% of respondents
• Half: findings reflect the views and opinions of 50% of respondents (“almost half” and “just over half” are also used when one less or additional respondent shares an opinion)
• Some: findings reflect the views and opinions of at least 25% but less than 50% of respondents
• A few: findings reflect the views and opinions of at least two respondents but less than 25% of respondents

Case studies

The case studies were intended to provide a source of information on lessons learned that could be used in subsequent HR modernization projects.

Six case studies were conducted, two for each of three components of the HRSM initiative (CHRBP, Human Resources Business Solutions Pilot (HRBSP) and PeopleSoft v9.1 Joint Build, and HRIS). The unit of analysis for each case study was the department, and the case study focused on the implementation, outputs and outcomes associated with that component. Other evaluation questions related to relevance and governance were also explored, again with the focus on the component at each department.

Each case study included a review of the documents and administrative data pertaining to the department and as well as the implementation and outcomes of the component for which they were selected. As well, interviews were conducted with up to two individuals, who had knowledge of the component and its implementation and outcomes, from each department.