Evaluation of the Classification Program

Ongoing need for the program

All lines of evidence confirm that there continues to be a need for the program. Most important, the program has an obligation to fulfill its stewardship role for classification in the CPA, as mandated by legislation. In addition, no other program allows the Government of Canada to determine and manage fair and competitive compensation for employees of the CPA.

As part of this foundational role, there is a pressing need to review the OGS and classification standards, and where warranted, to streamline and update the qualification and job evaluation standards, some of which are more than 50 years old.

Most survey respondents agreed that there continues to be a need for the program to provide DAs with support to implement policy and with expertise to inform grievances and complaint resolution. There is also a need for monitoring compliance with policy and for accrediting classification specialists. A few interviewees also highlighted the need for the OGS review to support priorities such as horizontal initiatives, an agile and mobile workforce, and flexible structures.

The document review confirmed that the program is required to meet commitments made to bargaining agents. Specifically, TBS entered into an agreement with the Public Service Alliance of Canada to review the occupational group structure for the Program and Administration Services (PA) Group, which is to be completed by the end of 2017 This information has been severed. TBS has also committed to the Professional Institute of the Public Service of Canada to undertake a review of the Computer Systems (CS) Group. And in the most recent round of collective bargaining, TBS committed to the Public Service Alliance of Canada to conduct a review of the classification standards for the Technical Services (TC) Group by the end of 2019.

Alignment with priorities, roles and responsibilities

All lines of evidence show a strong alignment between the program and federal government priorities and roles and responsibilities.

The program is well aligned with the role of TBS. Section 7(1) of the Financial Administration Act establishes the responsibility of the Treasury Board to act on “all matters relating to a) general administrative policy in the federal public administration; [and] b) the organization of the federal public administration or any portion thereof, and the determination and control of establishments therein….” Similarly, section 11.1(1), “Powers of the Treasury Board,” states that in the exercise of its human resources management responsibilities, the Treasury Board may “a) determine the human resources requirements of the public service and provide for the allocation and effective utilization of human resources in the public service; [and] b) provide for the classification of positions and persons employed in the public service.” The Public Service Employment Act provides Treasury Board with the authority to establish qualification standards. While deputy heads have accountability for their DAs, TBS is responsible for establishing policies, standards and associated guidance to ensure internal and external relativity.

Immediate outcomes

Expected outcome: DAs are supported and have the expertise to deliver classification program activities

Conclusion

The program is achieving this expected outcome to some extent. A lack of capacity appears to prevent the program from fully achieving this outcome. A significant proportion of DAs do not believe they have adequate expertise in their organizations.

The evidence on support to DAs from the online survey and key informant interviews is mixed. A slight majority of survey respondents responded positively about the program’s capacity to support DAs, whereas interviewees responded more negatively.

While 52% of survey respondents agreed that the program has the necessary capacity to support their organization in delivering classification activities, 21% disagreed with this statement and 27% were neutral. This finding was also supported by interview data.

Most interviewees said that the program appears to have limited resources to support DAs. Many activities conducted by the program required reassignment of resources (for example, work on special requests from DAs, support to implement unplanned proposals for updating occupational group definitions and standards, and support to the Pay Equity policy team). Some interviewees mentioned that this lack of resources prevents the program from being strategic.

Despite capacity challenges, the document review showed that the program has developed a series of tools to support DAs. These tools include:

• dashboard
• job evaluation plans
• learning curriculum for accreditation purposes
• information and reporting systems such as the Position and Classification Information System (PCIS) Query Tool and PCIS+ tools
• Job Title Abbreviation Tool
• Reclassification Proactive Disclosure on the Open Government Portal

A few respondents indicated that more work is needed to develop oversight tools, and tools for chief financial officers to help them to understand the financial implications of classification decisions.

Overall, evidence from interviews and the survey confirmed that most DAs believe they have adequate classification expertise (64% of survey respondents); however, another 33% disagreed with this statement.^{Footnote 5} This finding is also reflected in the 2016 Classification Monitoring Aggregate Report. Program documentation also indicates that DA capacity in classification expertise (including accredited advisors) varies across DAs.

Data from the 2016 classification dashboards indicate that 43% of all occupied positions in the CPA have job descriptions that are more than 5 years old, an increase of 3.7% since 2015. The Directive on Classification recommends that job descriptions be reviewed every 5 years. As shown in Figure 2, the number of outdated job descriptions has been steadily increasing.

The reported lack of capacity in approximately one-third of DAs may explain the lack of progress on standardized and updated job descriptions in DAs. While it was beyond the scope of this evaluation to measure the impacts of outdated job descriptions, there are risks associated with them. Outdated job descriptions could indicate that they do not match the actual content of the work, which can provide distorted information to human resources functions such as collective bargaining, pay, performance management and recruitment.

Figure 2 - Text version

The title of Figure 2 is “percentage of job descriptions in the CPA by age, March 2014 to March 2016.”

This figure is a historical illustration of the age of job descriptions in the CPA from 2014 to 2016. The figure is a 2‑D column graph. Categories are on the horizontal axis (x‑axis), and values are on the vertical axis (y‑axis). The horizontal axis shows the age of the job descriptions for 2014, 2015 and 2016 in 5 categories (0 to 5 years, 5 to 10 years, 10 to 15 years, 15 years or more, and no data available). The vertical axis shows the percentage of job descriptions for each category. The vertical axis starts at 0 and ends at 70, with ticks every 10 points.

The percentage of job descriptions between 0 to 5 years was 64.3% for 2014, 60.7% for 2015, and 57% for 2016.

The percentage of job descriptions between 5 to 10 years was 22.1% for 2014, 25.2% for 2015, and 27.6% for 2016.

The percentage of job descriptions between 10 to 15 years was 9% for 2014, 9% for 2015, and 9.6% for 2016.

The percentage of job descriptions for 15 years or more was 4.2% for 2014, 2.8% for 2015, and 5.6% for 2016.

The percentage of job descriptions for which no data are available is 0.4% for 2014, 0.3% for 2015, and 0.1% for 2016.

Training and development: The document review indicated that training activities have been designed and delivered for the classification community since 2016, namely for the accreditation of human resources advisors and the training of managers with human resources responsibilities. A new learning curriculum has been developed in partnership with the Canada School of Public Service. The new curriculum consists of four courses, and over 15,000 managers have received training. According to the survey findings, 68% of respondents agreed that course materials for classification-related training are effective for training managers and human resources advisors. Respondents from smaller DAs were more likely to agree (81% versus 60% for those from larger DAs).

Governance effectiveness in support of program efficiency

The Workforce Organization and Classification Directorate is responsible for designing and delivering the program, as well as for supporting DAs and monitoring classification decisions. The OGS Assistant Deputy Minister Steering Committee provides oversight and guidance for classification initiatives. The steering committee has a mandate to assess issues on the organization of work and to provide advice to OCHRO.

Most current and former committee members interviewed for the evaluation were satisfied with the steering committee’s mandate and activities. However, a few mentioned that the meetings are too short and not frequent enough. Some said that documentation stemming from the meetings was limited.  

The evidence suggests that the program could benefit from a governance structure that goes beyond the OGS. A few interviewees suggested that more senior-level committees would be useful to establish links with other OCHRO units. A committee with membership both internal and external to the Compensation and Labour Relations Sector could help set priorities. The document review also supported this finding.

A key informant mentioned that major obstacles to overall progress are the lack of consensus on the objectives for OGS reform, and the need for more work to establish long-term objectives that consider operational, legislative and labour relations constraints. Some said that the OGS needs to be more directly connected to other government-wide human resources initiatives.

The 2003 May Status Report of the Auditor General of Canada and successive program documents have called for the revision and modernization of the classification system. The evidence suggests that stronger governance could provide the vision, direction and support needed to address the backlog of work in a timelier manner. This governance could also encompass new commitments and provide direction, taking into account both the complexity of the issues and the changing environment in which the program operates.

Program use of resources

According to documentation, the program’s budgeted resources in fiscal year 2013 to 2014 included 27 full-time equivalents (FTEs) through B-base funding, in addition to 11 FTEs through A‑base funding, for a total of 38 FTEs. However the actual FTEs used in the past 4 fiscal years have fallen below this level (see Figure 3). Both program documentation and interviewees indicated that the program does not have sufficient human resources to produce the expected program outputs.

Figure 3 - Text version

The title for Figure 3 is “program resources by type of program funding, 2013 to 2014 fiscal year to 2016 to 2017 fiscal year.”

This figure is a historical illustration of the program resources from the 2013 to 2014 fiscal year to the 2016 to 2017 fiscal year. Categories are on the horizontal axis (x‑axis), and values are on the vertical axes (y‑axis on the left and z‑axis on the right).

This figure has stacked columns that show the sources of funding (A‑base and B‑base funding) in thousands of dollars, and a trend line that shows the actual number of full-time employees (FTEs).

The horizontal axis shows the fiscal years from 2013 to 2014 to 2016 to 2017. The first vertical axis (y‑axis) shows A‑base and B‑base funding. This axis starts at 0 and ends at 6,000, with ticks every 1,000 points. The second vertical axis (z‑axis) shows the number of actual FTEs. This axis starts at 20 and ends at 40, with ticks every 5 points.

From 2013 to 2014, A‑base funding was $1,208,000, B‑base funding was $1,541,000, and there were 27 FTEs.

From 2014 to 2015, A‑base funding was $941,000, B‑base funding was $4,215,000, and there were 32.6 FTEs.

From 2015 to 2016, A‑base funding was $1,099,000, B‑base funding was $4,263,000, and there were 36 FTEs.

From 2016 to 2017, A‑base funding was $1,095,000, B‑base funding was $4,354,000, and there were 35 FTEs.

Source: Classification Program administrative data

According to interviews and documentation, the program’s ability to produce the expected outputs (both in terms of quality and quantity) was hampered by four factors:

1. Understaffing: In 2015, securing qualified staff for the program was recognized as a key challenge in program reports (the program had 32.6 FTEs rather than the 38 approved FTEs). In 2016, staffing was again recognized as a challenge although the actual FTE count increased (36 actual FTEs). Similar challenges were noted for recruiting external resources. Some interviewees said that the program needs twice the resources it currently has.
2. Turnover among staff, including among senior management: Key informants mentioned that turnover put additional pressures on staffing. Many OCHRO senior managers had changed positions since 2011, resulting in delays in some ongoing projects, especially the OGS changes.

3. B‑base funding model: Most interviewees agreed that the program is ongoing in nature and should receive continuous funding. The program is mandated by legislation, and there are activities, such as oversight, which should be ongoing and funded permanently to prevent misclassification and upward trending in the classification structure. As well, many respondents agreed that by the time the OGS and job evaluation standards are updated in the current cycle, revisions will likely be required to the earlier work done by the Workforce Organization and Classification Directorate, given that the OGS structure (and related job evaluation standards) should be reviewed on a regular basis.

   B‑base funding is usually applied to short-term projects. The document review showed that the timelines for the Classification Program Renewal Initiative have extended beyond what was originally planned. This finding suggests that without additional resources, the modernization of the classification system may take longer than originally estimated.

   A majority of key informants interviewed stated that the program should be completely A‑base funded. They believe that the use of B‑base funding has led to inefficiencies. For example, staffing and retaining qualified personnel can become challenging since the short-term arrangement implied by B‑base funding can create uncertainty about job stability for some employees. In addition, a number of interviewees mentioned that producing the biannual progress reports for Treasury Board, which is a requirement for B-base funding, presented an excessive administrative burden.

4. Unexpected and unfunded tasks: Such work prevents the program from producing its expected immediate outcomes. This information has been severed:
   1. Policy development: After the planned overhaul of the Classification Policy Suite, completed in 2015, the Workforce, Organization and Classification Directorate also redesigned its Policy Suite in 2016 to align with the new approach of the Policy Reset Initiative (Policy on People Management).
   2. Learning and capacity building:In 2016, the Canada School of Public Service changed its delivery model. As a result, the program is now responsible for all new and updated course content, the identification and training of new facilitators, the correction of exams, and classification accreditation that involves extensive documentation reviews.
   3. Machinery of government changes: The government’s decision to transfer certain organizational components of the Royal Canadian Mounted Police (RCMP) to the CPA required analysis to determine how to integrate staff in non-CPA job classifications into the OGS. This led to the analysis and determination of acquired rights for RCMP civilian members and the design of the Police Operations Support (PO) Group. Similarly, in the 2012 to 2013 fiscal year, the consolidation of CS staff into the newly created Shared Services Canada required the provision of policy guidance for staff transfers.
   4. Pay equity complaint litigation:Between 2012 and 2017, the program provided support for several major pay equity litigation cases that exceeded the resources allocated for litigation management. For example, the program developed a customized classification standard and methodology for a pay equity case to be able to compare and evaluate the FI‑01 and FI‑02. For this case, 152 groups and levels were evaluated This information has been severed. Program resources were required to support the Public Sector Equitable Compensation Act policy and legislation efforts
   5. Enterprise-wide initiatives: The following initiatives were supported by the Workforce, Organization and Classification Directorate over the period under evaluation:
      1. The program’s contribution to the redesign of the Organization and Job Classification component of the Common Human Resources Business Process, and support for its implementation across the CPA.
      2. The shift to the new Canada.ca website, which involved migrating existing web content and TBS intranet pages, sunsetting the former Position Reclassification Data Capture tool used for Proactive Disclosure, and helping departments migrate to the new Open Data Portal to disclose their reclassifications.
      3. Expanded work on the PCIS+ system: Initially, the plan was to build just a Work Description Repository. However, the PCIS Query Tool was an aging legacy system that needed to be upgraded and both systems would use very similar data. In line with the Clerk of the Privy Council’s priority to “streamline the back office,” it was decided to combine the two tools. The program was identified as the pilot sector for building a new master database, with PCIS data to be used by all TBS applications in the future.
   6. Other OGS work:Priorities for the OGS review project were changed twice at senior management direction to:
      1. modify the Foreign Service (FS) Group definition This information has been severed
      2. review all job evaluation standards to reflect the government’s approach to its relationship with Indigenous Peoples, and revise six standards to align with the reconciliation initiative

Impact on government-wide efficiency

To date, work on the OGS is ongoing but has not yet resulted in a measureable government-wide efficiency This information has been severed. Changes are not cost-neutral and normally take time to produce cost savings. In fact, OGS changes frequently lead to cost increases for government in the short term, while benefits are indirect and longer term.

For example, a new job evaluation standard means new rating factors for valuing jobs. Job evaluation using new factors usually means that some jobs will be worth more than under previous standards and some will be worth less. Where jobs are worth a higher salary level, the incumbents will be compensated; where the jobs are worth a lower salary level, the incumbents have access to salary protection and do not experience a reduction in pay. In the longer term, classification reform provides important benefits to creating an agile, mobile workforce, but it requires updated classification standards and an updated OGS.

Relevance

1. How has the Classification Program (the program) adapted to its changing context over the last five years, and what are the implications for the program?
2. How does the program align with the roles and responsibilities of the Treasury Board of Canada Secretariat (TBS)?
3. To what extent does the initiative continue to address a demonstrable need?

Performance

4. To what extent has the program achieved its immediate outcomes?
5. What are the factors that help or hinder the achievement of outcomes?
6. Is the right governance model in place to oversee and manage the program?
7. How effective is the program oversight model?
8. How effective is the current classification model in the Government of Canada (that is, building capacity within TBS and departments and agencies (DAs) to address occupational group structure (OGS) changes, learning and accreditation)?

Efficiency

9. How could the program be more cost-efficient?
10. To what extent is the program able to achieve outcomes within its current level of resourcing?
11. Are there delivery alternatives that could decrease costs while delivering similar results?

Alternatives

12. What lessons have been learned to date, or what best practices have been applied?
13. Are there any barriers or unintended outcomes that have been observed?

Analysis of administrative data

The analysis of program administrative data that is maintained by the Classification Program managers includes the analysis of the data collected for regular reports prepared for the Public Service Management Advisory Committee and other core public service audiences, and for the day-to-day management of the program. Data from the Position and Classification Information System (PCIS) was also analyzed. Finally, data maintained by the Canada School of Public Service on participation in, and completion of, classification classes was analyzed.

Document review

The document review provided evaluation evidence to supplement and expand on the analysis of the administrative data. The documents included plans and frameworks, progress reports, dashboard reports, third-party assessments, memoranda of understanding, lessons learned documents, research reports, monitoring reports and results, audit reports and results, agendas and minutes from governance bodies, applicable regulations and policy instruments, the 2003 May Status Report of the Auditor General of Canada, information on the program published on GCPedia and Canada.ca, and briefings and presentations to various audiences.

Literature review (review of practices and structures of other jurisdictions)

The evaluation team also conducted a review of the practices and structures of other jurisdictions, including Ontario, Quebec and Australia. The team also reviewed practices in another service sector in the federal government (legal advisory services) as alternative approaches to delivering the program.

Key informant interviews

Twenty-five in-depth key informant interview sessions were conducted using open-ended questions. Key informant interviews are a qualitative method used in evaluation to address most evaluation issues and questions. They gather views and factual information from key informants selected from within the federal government and the core public administration.

Interviews were conducted with representatives of:

• the Classification Program, including the Director, managers and other key staff members with relevant information to inform the evaluation questions (n=6)
• the OGS Assistant Deputy Minister Steering Committee that guides the work of the Classification Program (n=5)
• the Human Resources Council and other applicable communities, including functional communities such as the Organization and Classification Community of practice (n=6)
• the Public Service Commission of Canada or departmental heads of staffing (n=2)
• representatives from other sectors at TBS
  • Government Operations Sector (n=2)
  • Expenditure Management Sector, Compensation Management (n=2)
  • Corporate Services Sector (n=2)

When invited to participate in an interview, respondents were invited to organize group interviews, if they wished to do so. Each interview was weighted equally regardless of the number of interviewees in each interview.

Online survey of DAs

Evidence from DAs was gathered via an online survey conducted in April and March 2017. The survey was sent to 112 heads of classification representing DAs of the core public administration. A total of 69 individuals responded to the survey, for an overall response rate of 62%. Several email reminders were sent to the potential respondents, and follow‑up phone calls were made to those who did not respond.

The following scale was used to summarize the survey results:

• few: less than 20%
• some: 20% to 39%
• many: 40% to 49%
• half: 50%
• most: 51% and more