Formative evaluation of Public Service Commission-led Pools

Final report

Evaluation Division
Corporate Management Practices and Evaluation Directorate
Corporate Management Branch
Public Service Commission

May 25, 2010

Table of contents

• Executive summary
• 1 Introduction
• 2 Methodology
  • 2.1 Evaluation issues
    • 2.1.1 Relevance
    • 2.1.2 Performance
    • 2.1.3 Cost-effectiveness
• 3 Profile of Public Service Commission-led pools
  • 3.1 Background
  • 3.2 Characteristics
  • 3.3 Governance
  • 3.4 Creating and administering Public Service Commission-led pools
  • 3.5 Stakeholders
• 4 Logic model
• 5 Findings and analysis
  • 5.1 Relevance of Public Service Commission-led pools
    • 5.1.1 Priorities of the Government of Canada and the Public Service Commission
    • 5.1.2 The need for Public Service Commission-led pools
  • 5.2 Performance of Public Service Commission-led pools
    • 5.2.1 Client expectations
    • 5.2.2 Governance and accountability
    • 5.2.3 Processes and mechanisms
  • 5.3 Cost-effectiveness of Public Service Commission-led pools
    • 5.3.1 Cost recovery results
  • 5.4 Alternative business models
• 6 Summary of recommendations and management response
• 7 Management action plan
• 8 Limitations of the evaluation
• Appendix A: Client and non-client organizations interviewed
• Appendix B: Public Service Commission-led pools, January 2010
• Appendix C: Public Service Commission-led pools Logic model
• Appendix D: Comparative analysis - Collective staffing case

Executive summary

What was examined

The evaluation was carried out for the President and the Executive Management Committee of the Public Service Commission (PSC), as part of the PSC's Risk-Based Five-Year Evaluation Plan 2009-2014. This report presents the findings for the evaluation of PSC-led pools. The evaluation is formative (i.e. it was designed to be conducted during the development and early implementation phases of PSC-led pools). It was done primarily for the purposes of examining whether, or the degree to which, changes or modifications are required for PSC-led pools to help them perform better, and to examine their alignment with the priorities, goals and objectives of the PSC and the Government of Canada.

The stated objective of the PSC-led pools was to increase the effectiveness and efficiency of specifically targeted external staffing actions. The PSC has recognized that positions with similar requirements exist across federal organizations (e.g. administrative and clerical positions) and created PSC-led pools to respond to staffing needs for these positions. Candidates from PSC-led pools are referred to client organizations for a fee and in some instances fees are charged according to the number of candidates hired.

Why pools are important

PSC-led pools are a new and innovative way for the PSC to fulfill its role as a common service provider and to complement other PSC services, namely staffing, assessment and other pools of pre-tested candidates, such as the Post-Secondary Recruitment Program. With pools of assessed candidates for mainly administrative and clerical positions, the PSC is able to allow government organizations to staff positions quickly and efficiently. A formative evaluation of PSC-led pools was conducted to recommend improvements to PSC-led pools and help them maintain and improve their overall performance, including cost-effectiveness, design and success. Key stakeholders and beneficiaries of PSC-led pools are client organizations and applicants. PSC-led pools were created in an effort to reduce the time to staff across government and could contribute to reducing the total number of job advertisements government-wide.

What we found

PSC-led pools do not, however, have consistent, clearly defined goals and objectives related to service delivery, business processes, needs assessment and client satisfaction. This situation has resulted in inconsistent service delivery, as well as possible overlap with other PSC staffing services. A more consistent, accountable and appropriate management structure is recommended to support ongoing stability, accountability and improvements.

Overall, there is an inherent risk in PSC-led pools where there is no assurance that costs will be recovered, as may be the case with other PSC cost-recovery products and services. Costing strategies should be reviewed to help mitigate these risks. Additionally, there is potential overlap between PSC-led pools and collective staffing in other federal organizations; however, there is still a need for a government-wide initiative.

In an effort to provide concrete information to help support successful PSC-led pools in the future, the evaluation team identified certain good practices that could be considered when additional pools are developed. These include the following:

• A thorough needs analysis that factors in high-demand job classifications;
• Bilingual pools where candidates have been pre-assessed in writing and reading in second language evaluation areas;
• The level of assessment should be aligned specifically with client needs;
• The size of the pool relative to regional demand (e.g. a pool should not be closed before all clients have hired their candidates or before there is an insufficient number of candidates remaining in the pool to meet client demands);
• The breakeven point on a pool can be realized sooner if the pool is begun earlier in a fiscal year. However, there may be a need to initiate pools in the previous year in order to have candidates available in the first quarter of the fiscal year, when recruitment activity is highest; and
• Regions should routinely verify candidates' interest and be more time sensitive to client needs when making referrals.

What we recommend

The PSC should anticipate and map out clients' needs through ongoing communications and environmental scanning and factor in high-demand regions and high-volume job classifications (AS and CR streams). Particular attention could be paid to niche segments. Efforts should be made to target small federal organizations, positions in remote locations and those regions without a human resources presence, where needs are potentially the greatest. PSC-led pools should promote their services more widely to increase market penetration. Post-sale data, such as the time federal organizations take to staff a position once they have received a candidate, should be systematically collected.

The PSC should develop uniform service delivery processes and standards to address service-related issues promptly and effectively. To identify areas for improvement, the PSC should centrally monitor satisfaction levels among candidates in PSC-led pools, as well as other relevant performance measures, such as the length of time from client requests to the delivery of referrals. The PSC should also implement formalized goals and objectives, operational guidelines and a risk-mitigation strategy to enable PSC-led pools to reach their full potential.

The PSC should identify a lead and clarify roles and responsibilities related to PSC-led pools, while maintaining their flexible nature.

The Staffing and Assessment Services Branch and, in particular, the Integrated Services Directorate should consider conducting a costing review to define a consistent and clear methodology and reporting process for all regions. This cost-reporting system should ensure that all actual recoverable costs related to PSC-led pools are accurate and comparable among regions.

1 Introduction

This report presents the findings of the evaluation of PSC-led pools, a human resources staffing product for federal organizations. PSC-led pools are established with an anticipatory staffing perspective without formal documentation such as a service agreement. Each of the PSC-led pools contains a number of candidates who have applied to a particular advertisement on jobs.gc.ca and who meet the minimum criteria for that staffing pool.

The evaluation was carried out for the President and the Executive Management Committee of the Public Service Commission (PSC), as part of the PSC's Five-Year Evaluation Plan 2009-2014. Additionally, the Policy on Evaluation, 2009 of the Treasury Board Secretariat calls for the relevance¹ and performance² of policies, programs and initiatives to be examined. The evaluation was formative; it was carried out during the development and implementation phases of PSC-led pools. For this reason, the two main aims of the evaluation were to determine whether changes are needed for PSC-led pools to meet their stated objectives, and to offer information that can make PSC-led pools a better managed human resourcing mechanism. The evaluation also examined cost-effectiveness as part of evaluating performance.

The evaluation was undertaken by the Evaluation Division of the Corporate Management Practices and Evaluation Directorate in the Corporate Management Branch.

2 Methodology

The evaluation took place between August and November 2009. Additional financial data from January 2010 were incorporated as they became available. The work involved the following tasks:

• Reviewing documents and files; creating a logic model and a gap analysis for each stream of the logic model; developing a risk analysis; and producing an evaluation matrix;
• Preparing methodological strategies, interview guides and questionnaires; creating a database, collecting and analyzing financial and administrative data; conducting a comparative analysis using case studies of collective staffing processes at the PSC and in other federal organizations (Transport Canada, Agriculture and Agri-food Canada and Human Resources and Skills Development Canada); and
• Surveying candidates in the pools and hiring managers across the Government of Canada.

To help identify and confirm issues covered by the evaluation, the team interviewed client and non-client hiring managers, regional operations staff and PSC managers at headquarters. The list of federal organizations interviewed is found in Appendix A.

The design of the logic model, evaluation matrix, interview guides and two survey instruments was based on the document and data review and validated by management of the PSC-led pools.

2.1 Evaluation issues

This section describes the issues and questions raised by the evaluation.

2.1.1 Relevance

To be relevant, PSC-led pools should be compatible with the priorities of the PSC and the government as a whole, and should be needed on an ongoing basis. Two central questions were identified:

1. To what extent are the objectives of PSC-led pools consistent with the policies and priorities of the Government of Canada and the PSC?
2. How big is the need among federal organizations for a centralized pool management function offered as an optional service? Can PSC-led pools evolve with the changing environment to maintain their relevance? Is there duplication of other governmental efforts?

2.1.2 Performance

The evaluation team examined performance from the point of view of various stakeholders, including hiring managers and program managers. These stakeholders answered questions about access, usage, satisfaction, design, communications and effectiveness. Three questions about performance were identified:

1. To what extent do PSC-led pools meet client expectations?
2. To what extent do the governance and accountability structure and the business model support the objectives of the pools? Are processes and mechanisms (including communications efforts) working as intended?
3. To what extent are the staffing needs of federal organizations appropriately targeted?

2.1.3 Cost-effectiveness

The evaluation looked at how successful PSC-led pools were in offering clients a cost-effective solution to staffing. Two questions about cost-effectiveness were identified:

1. Are PSC-led pools cost-effective? Are certain types of pools more cost-effective than others?
2. Does the current costing and charging structure effectively reflect costs?

3 Profile of Public Service Commission-led pools

3.1 Background

PSC-led pools were started in January 2008 in response to an identified need for pre-assessed candidates on a government-wide basis. Based on information provided by the initiative, their objective is to offer faster, more effective and more efficient staffing for specific classifications and levels through external processes. Program staff point out that the PSC is uniquely positioned to deliver cost-effective services to federal organizations across the country because of its national presence and expertise. The PSC helps federal organizations meet their human resources needs by providing advice and consulting on staffing activities, offering such services as e-resourcing, advertising, screening, student and graduate recruitment programs and collective staffing.³

The PSC created PSC-led pools to help federal organizations fill positions with common characteristics, such as administrative and clerical positions among others, quickly and easily. It is one staffing tool among many that help the PSC to fulfill its mandate as a "leader in public service staffing and assessment."⁴ However, some organizations, usually larger ones, can and do develop their own pools for their internal needs.

To differentiate PSC-led pools from collective staffing, PSC officials are using a working definition that PSC-led pools do not use formal documentation such as master service agreements and memoranda of understanding with clients. Rather, client organizations pay a fee of $1 800 to receive up to five referrals of partially assessed candidates. Fees charged for fully assessed candidates are dependent on the assessments carried out and average $2 500 for each hire. The fees are collected as part of the PSC's cost recovery program.

3.2 Characteristics

Since April 1, 2008, 43 pools and four inventories⁵ have been set up: non-assessed pools, various levels of partially assessed pools and fully assessed pools. From an inventory, the PSC can start pools as they are needed.

Because each regional market is different, pools vary in size, length of operation and level of assessment (including second language evaluation). In January 2010, 25 pools and one inventory were managed by the PSC. From April 2008 to June 2009, the PSC-led pools included 21 745 unique candidates.⁶ From the beginning of the pools to January 2010, approximately 1 000 partially and fully assessed candidates were referred to hiring organizations, mostly in Clerical (CR) and Administrative (AS) positions.⁷ Approximately 22% of assessed candidates have been referred to date. A complete list of existing PSC-led pools is available in Appendix B.

3.3 Governance

PSC-led pools are not an official program and have only a limited governance structure: the initiative has no budget, no specific internal PSC-led pools policy, no statement of goals and principles, no organizational chart and no other building blocks that define a public service program.

There is no specific entity that governs the overall performance of PSC-led pools. Pools are carried out as a shared function. At the start of the initiative, the idea was to make the Integrated Services Directorate the financial centre; however, due to resource constraints, the idea was not developed further. The Operations Directorate of the Staffing and Assessment Services Branch (SASB) liaises with the regions and other SASB services, as well as the Director General of Operations, who is responsible for the administration of PSC-led pools in the regions (e.g. co-ordinating candidate lists, client contact, referrals). The National Capital Region reports to the Director General of National Client Services Directorate.

Figure 1 - SASB organizational chart

Figure 1 - long description

PSC-led pools are subject to the PSC's cost recovery system. Each year, the PSC's Integrated Services Directorate sets the fees for referrals (and, in some cases, hires) to federal organizations. These fees apply nationally; the regional offices do not have the authority to change them. Regional offices provide Staffing Operational Services with monthly appointment and referral updates.

3.4 Creating and administering Public Service Commission-led pools

Federal organizations contact the PSC regional office to determine if there is a pool for the criteria and candidates they seek. Requests for pools can also be received through input from federal councils. If such a pool exists, the federal organization may submit a request to the PSC regional office, stating the type of employment (indeterminate or term), duration, as applicable, work location and Statement of Merit Criteria (SoMC) for the position to be filled.

If a pool for that particular position does not exist, the regional office begins a human resources needs analysis. Systematic needs assessments have not been part of standard operating procedure and have not been done consistently in all cases. Once the need for a pool is determined, the region outlines the SoMC, level of assessment, and the ideal size and duration of the pool, based on their experience.

Once a PSC-led pool is established, the regional office is responsible for posting advertisements on jobs.gc.ca, receiving and screening applications, assessing applicants, referring candidates to hiring organizations, and maintaining accurate databases of various pools. In most cases, five candidates are chosen at random and referred to a hiring organization at its request. In most cases, the hiring organization interviews candidates and can do further assessments. In some instances, the PSC continues the assessment at the request of the client organization for an additional fee based on a per diem rate. PSC regional offices decide when and if a pool should be increased, maintained or closed. PSC-led pools generally expire when there are an insufficient number of candidates remaining in the pool to meet client demands, the assessments are no longer valid or there is no more interest from hiring organizations.

Partially assessed pools are most common. They generally include screening for experience and assessment using PSC standardized tests. Depending on the pool, assessments include any combination of testing, language assessment, interviews, reference checks and screening for experience.

3.5 Stakeholders

Stakeholders and others with an interest in PSC-led pools include the following:

Job seekers

Job seekers have a strong interest in effective and efficient PSC-led pools. When they apply to advertisements on jobs.gc.ca, their names are put into a database and they begin the screening phase of the process. They become candidates for jobs in the public service after they are screened into a pool or inventory. In January 2010, there were 4 592 assessed candidates⁸ in PSC-led pools (excluding the non-PSR inventories, which have 10 671 candidates).

Survey results

For the purpose of the evaluation, a survey of 21 745 candidates to PSC-led pools was conducted in September 2009 and a profile was created from the 9 425 responses (a response rate of 43%). A summary is detailed below:

• 70% of survey respondents are female and 76% are English-speaking;
• 31% are members of visible minorities, 5% are Aboriginal persons and 5% are persons with disabilities;
• 46% have university degrees;
• 69% of respondents felt that there are government jobs in their region;
• 60% of respondents felt that PSC-led pools provide Canadian job seekers with access to job opportunities in the public service; and
• 15% of respondents were unemployed, 41% were employed in the private sector and 22% were employed in the public sector when they entered a pool.

Public Service Commission's regional and operational managers

Much of the success of PSC regional offices depends on their ability to provide value-added services to federal organizations. Regional personnel analyze the potential staffing needs of their clients based on historical data to determine the types of pools needed. When pools are effective, efficient and result in quality referrals to client organizations, the reputation of the regional office is directly enhanced.

The Public Service Commission

The PSC has a strong stake in a well-administered PSC-led pools system. It co-ordinates and develops client-centred products and services to make staffing positions a more effective and efficient process. Developing innovative tools such as PSC-led pools permits the PSC to maintain and build its reputation as a human resources partner that offers a "full range of services and tools [to] help tailor customized solutions to each client's specific needs."⁹ When clients perceive PSC-led pools as useful and successful, the PSC's contribution is further recognized.

Client organizations

Hiring managers at client organizations are direct stakeholders of PSC-led pools. They are interested in accessing efficient, streamlined staffing products and services that provide quality candidates who have been assessed promptly. A needs assessment of the staffing requirements of clients is the main driver for creating a pool. Regional and federal councils also play an important role in determining the need for PSC-led pools.

Canadians

As a government-wide initiative, PSC-led pools allow Canadian job seekers access to federal employment and permit them to apply to a central database and be considered for jobs in any federal organization governed by the Public Service Employment Act (PSEA).

When PSC-led pools work well, they help to maintain and enhance the reputation of the Canadian public service in the minds of Canadians. In particular, the PSC is seen as an agency that embodies its stated values of integrity, fairness, respect and transparency.¹⁰

4 Logic model

A logic model is a visual representation of the inputs, activities, outputs and outcomes of an initiative. It is attached as Appendix C.

The logic model was analyzed to identify strengths and weaknesses of PSC-led pools based on the projected goals. They are listed by activity stream. As PSC-led pools are fairly new and still evolving, the operational team is learning while doing and trying to minimize the gaps between how PSC-led pools should operate and how they actually function. A summary of these gaps includes the following:

Communications and outreach

• The biggest gap found in this stream is in the relationship between job seekers and PSC-led pools. Survey results suggest that job seekers have limited awareness and understanding of the procedures for PSC-led pools. Candidates surveyed felt that the PSC did not keep them well informed of their status in a PSC-led pool (66%).

Needs analysis

• Environmental scans, needs analysis and recognition of business development opportunities must be started in some regions and formalized in others. Business case proposals for each pool have to be developed systematically. At the moment, these activities are conducted in an ad hoc fashion.

Assessment and service delivery

• Since service delivery is core to PSC-led pools, the operational team has focused most of its efforts and resources in that area. However, there are still some gaps in how activities are carried out, particularly in providing feedback to job seekers and candidates. These activities seem to be the strongest area of PSC-led pools.

Management of the initiative

• Business metrics and other data sources, such as management information processes, are key tools for assessing and measuring performance and results. As of November 2009, performance measurement data range from limited to inadequate, and standardized national procedures do not exist. This situation creates complexity in assessing success.

5 Findings and analysis

The following section presents the findings of the evaluation team organized around the themes of relevance, performance and cost-effectiveness using the evaluation questions. Conclusions are drawn on major themes and sub-themes as appropriate. Recommendations are made only when the evaluation concludes that management should act.

5.1 Relevance of Public Service Commission-led pools

5.1.1 Priorities of the Government of Canada and the Public Service Commission

To what extent are the objectives of PSC-led pools consistent with the policies and priorities of the Government of Canada and the PSC?

In light of the focus on renewal of the public service, the Clerk of the Privy Council created a four-point Public Service Renewal Action Plan: planning, recruitment, employee development and enabling infrastructure. The Clerk pointed out in the 2008-2009 Action Plan that the government needs to "hire the right graduates and experienced workers to meet [the] varied work demands, as much as possible through direct recruitment"¹¹ mechanisms.

The Common Services Policy of the Treasury Board Secretariat sets strategic direction for federal organizations by streamlining administrative processes and saving costs through the use of centralized government-wide services, thereby aggregating supply in delivering services that are common to federal organizations. PSC-led pools as a hiring tool support the Policy through the provision of centralized government-wide expertise in delivering quality candidates.¹²

The PSC is the guardian of the merit principle and is responsible for implementing the PSEA. The Act states that the PSC "has the exclusive authority to make appointments to or from within the public service."¹³ While much of the staffing function has been delegated, staffing processes can be done collaboratively to meet the needs of federal organizations, while allowing them to benefit from time and cost savings. As a common service provider, the PSC takes advantage of economies of scale through its established role, expertise and investments in resourcing services and technology across Canada.¹⁴

A key priority for the PSC is to provide integrated and modernized staffing and assessment services¹⁵ that link Canadians with public service jobs. Using client service units across Canada helps to make the hiring process more efficient and increases access for Canadians. PSC-led pools "provide managers with an efficient and flexible way of staffing when there is a recurring need to fill similar positions in the same occupational group and at the same level"¹⁶ Such pools also "provide candidates with improved access to job opportunities, while protecting the hiring values of merit, fairness and transparency."¹⁷

This evaluation team found that the creation and administration of PSC-led pools are consistent with the priorities of the Government of Canada and with the goals of the PSC. PSC-led pools provide efficient and effective human resources services to federal organizations and are relevant in light of the government's Common Services Policy and ongoing efforts to streamline public service staffing. PSC-led pools are being developed in full compliance with the merit principle.

5.1.2 The need for Public Service Commission-led pools

How big is the need among federal organizations for a centralized pool management function offered as an optional service?

The needs of hiring managers

In the Sixteenth Annual Report to the Prime Minister on the Public Service of Canada,¹⁸ the Clerk states that "current human resources, business processes and systems do not … facilitate collective staffing and/or … pools of candidates, which could be considered in filling future needs." The Clerk's Report also calls for "a Public Service that is able to take risks, enhances productivity and removes barriers to efficiency and innovation."¹⁹ As the business of government becomes more complex, it requires new products and services, such as PSC-led pools, to help deal with these challenges.

In interviews, hiring managers reported that PSC-led pools are a useful tool that speeds up the assessment portion of the staffing process. Surveys of managers who had used the PSC-led pools showed that 45% were satisfied with the time to staff. When clients make their request to a PSC regional office, PSC-led pools expedite steps in the process, such as advertising the position, screening applications and performing assessments. However, it should be noted that hiring managers interviewed expressed a strong desire to see staffing times reduced further. In addition, 30% of managers surveyed were dissatisfied with the time to staff. As a result, the evaluation team concluded that more attention and performance tracking are required in this area.

The needs of small federal organizations

In interviews, both PSC operational managers and hiring managers in smaller organizations stated that PSC-led pools are a particularly useful option that saves time. Their hiring needs are limited and, in most cases, they cannot easily create their own pools. Small organizations benefit from having suitable candidates available -- especially, as managers noted in interviews, when candidates need little or no further assessment.

Hiring managers in federal organizations in isolated locations also found it easier to staff positions than was the case previously when candidates selected remote sites as a possible work location. In interviews, they stated that the National Area of Selection permitted them access to a wider range of candidates.

The needs of other federal organizations

Interviews found PSC-led pools to be an important and relevant tool that complemented the range of products and services offered by SASB. Findings suggest that PSC-led pools can be particularly useful to clients in large federal organizations with few human resources advisors in the regions as the pools provide needed capacity. PSC-led pools have made 1 000 referrals since they began.

The survey of hiring managers found that PSC-led pools are client-focused and have helped raise the PSC's profile. However, only 24% of surveyed hiring managers were very familiar with PSC-led pools, with almost 60% indicating that they had low to no familiarity with pools.

To date, the majority of pools have been undertaken in the regions. As the following table illustrates, the majority of the PSC-led pool clients are focused on mainly clerical and administrative positions (Tables 3, 4 and 6).

Several reasons have been identified for this strong regional focus for pools, ranging from the suggestion that federal organizations' HR capacity in the regions is limited to the fact that SASB analysts in the regions had the capacity to organize pools. From an NCR perspective, it was noted from the interviews that large organizations are often more likely to carry out their own processes and collective staffing as they have the capacity.

The needs of job seekers

Overall, PSC-led pools met the needs of job seekers but several areas were identified for improvement. Applicants to PSC-led pools benefit from its one-stop approach: one job application gives them access to a number of job opportunities. Candidates are informed that they may be referred to a hiring organization to help them find employment more quickly and easily. PSC-led pools thereby reduce the pressure on applicants by allowing them to apply for many public service positions through one channel and one application.

However, a survey of PSC-led pool candidates conducted in September 2009 for the purpose of this evaluation revealed four important reasons why they have relatively low satisfaction levels:

• The need for more communication about their status in pool(s) (66%);
• The often perceived or real delay between an application and a job offer (42%);
• The reported time-consuming application process (29%); and
• The uncertainty that candidates will receive a job offer at all (comments from survey)

While survey comments demonstrated that applicants appreciated the one-stop process, they also expressed some discomfort in the above areas. Therefore, PSC-led pools should implement business practices that allow for the collection of post-sale data, such as hiring managers' satisfaction levels and the time federal organizations take to complete the staffing process.

Can PSC-led pools evolve with the changing environment to maintain their relevance?

PSC-led pools are evolving to meet the needs of clients and candidates. Since the evaluation began, operational managers have begun a more systematic tracking of pool information to help manage performance, including information on referrals, hires and some financial data. As mentioned earlier, it was clear that systematic needs assessments were not routinely carried out prior to beginning a pool. Overall, these findings suggest that more attention is required to better understand client needs prior to establishing pools.

Outreach mechanisms should reach stakeholders to increase awareness of PSC-led pool activities and their contribution to the hiring process, such as reducing the length of the hiring process and communicating applicant status in a pool.

Is there duplication with other governmental efforts?

Sub-optimization of Public Service Commission-led pools

The evaluation team found that federal organizations are running staffing activities similar to what PSC-led pools have to offer in terms of classifications and merit criteria. However, this should not be considered as duplication given that organizations have delegated staffing authority. This may suggest that additional marketing efforts are needed to promote PSC-led pools as there is clearly a need for this type of initiative (see table below) and an opportunity to establish a streamlined and common approach that can generate economies of scale. Therefore, a centralized approach to staffing similar positions makes sense from a government-wide perspective.

A large number of staffing processes are posted every year for positions with similar classifications and merit criteria. The table below shows the number of external jobs posted in 2007-2008 for seven large federal organizations.²⁰

PSC-led pools currently offer pre-assessed candidates in these job classifications and, when used by federal organizations, offset the need for additional staffing processes. Managers of PSC-led pools stated that through an increased supply of candidates and economies of scale, they could contribute to lowered staffing costs by reducing staffing processes for particular job classifications, namely, CR and AS.

The evaluation team found a generally low level of awareness of PSC-led pools among hiring managers (50%), suggesting that increased information would contribute to greater optimization of PSC-led pools. In interviews and surveys, hiring managers saw real value in time savings among the other benefits of PSC-led pools. Several hiring managers interviewed and 64% of those surveyed stated satisfaction with the quality of candidates and 51% were satisfied with the ease of the hiring process through PSC-led pools.

Targeted marketing (i.e. identifying potential growth areas through forward-looking marketing strategies such as analyzing opportunities to provide staffing services in specific organizations) of PSC-led pools to hiring managers could potentially minimize the likelihood of future duplication. When hiring managers and HR advisors are aware of existing PSC-led pools, the need for other staffing mechanisms diminishes.

Evidence indicated that some PSC products without associated costs to hiring managers are drawing potential clients away from PSC-led pools, suggesting the need for a co-ordinated marketing strategy for PSC and SASB services. A hiring manager from a large organization stated that an easy way to minimize costs was to hire a candidate through the Post-Secondary Recruitment (PSR) Program, which does not charge for referrals. While this may not be a common practice, it suggests that managers are looking at ways to reduce spending while having access to pre-assessed candidates.

Even within the PSC, similar staffing activities occur. For example, job seekers with degrees often apply to both the PSR Program and PSC-led pools, with potential for assessment and administrative duplication. However, there are opportunities to maximize the use of the PSR inventory by using its candidates in the PSC-led pools. In that sense, both initiatives can be seen as complementary. While it can be argued that this range of staffing processes is designed to offer hiring managers in federal organizations their choice of products and services, there is a potential for redundancy and confusion among hiring managers as to the advantages and disadvantages of PSC staffing products and services.

5.2 Performance of Public Service Commission-led pools

5.2.1 Client expectations

To what extent do PSC-led pools meet client expectations?

Client expectations were evaluated by collecting satisfaction data on the quality of candidates, the cost of hiring, time to staff and ease of the hiring process. Between January 2008 and September 2009, 21 745 candidates were registered in PSC-led pools and 1 000 referrals were made to hiring organizations. As mentioned earlier, 24% of hiring managers surveyed were very familiar with PSC-led pools and 30% had used the initiative. Of those familiar with the pools, satisfaction rates with the quality of candidates were reasonable at 64%. Only 9% expressed dissatisfaction with the quality of candidates. Hiring managers were also more satisfied (51%) than dissatisfied (22%) with the ease of the PSC-led pools process. Section 5.2.3 addresses time to staff in greater detail.

In interviews, hiring managers said that PSC-led pools are a good alternative to running their own staffing process, but that relying on them involves some real or perceived disadvantages. These are as follows:

• Pools with fully assessed bilingual candidates often cannot supply enough candidates. Moreover, language skills of candidates in some partially assessed bilingual pools are often not assessed to client requirements (i.e. candidates are assessed on only two of three language areas).
• Clients sometimes experience a time lag without qualified candidates while waiting for the pool to be developed and for the initiative to collect information.
• Clients requiring specialized skills have more confidence in their departmental processes.
• The number of referrals available to clients can be inconsistent (i.e. clients usually receive up to five referrals, but that can vary depending on the size of the pool and availability of referrals).

PSC-led pools lack standardized processes in regional operations and service standards. Presently, client organizations are not given the maximum expected wait times for referrals or other guidelines. The PSC does not systematically collect client feedback on the quality of service or referrals.

Recommendation 1: Clients' needs and outreach

The PSC should anticipate and map out clients' needs through ongoing communications and environmental scanning and factor in high-demand regions and high-volume job classifications (AS and CR streams). Particular attention could be paid to niche segments. Efforts should be made to target small federal organizations, positions in remote locations and those regions without a human resources presence, where needs are potentially the greatest. PSC-led pools should promote their services more widely to increase market penetration.

Outreach mechanisms should reach stakeholders to increase awareness of PSC-led pool activities and their valuable contribution to the hiring process, such as reducing the length of the hiring process and communication of applicant status in a pool, in an effort to increase potential participation.

5.2.2 Governance and accountability

To what extent do the governance and accountability structure and the business model support objectives?

The regionally based approach currently used to manage PSC-led pools allows for flexibility in responding to client needs. However, their lack of formal structure and co-ordination increases the risks of PSC-led pools as they limit regions from sharing and benefiting from good practices and lessons learned. Furthermore, it does not permit the development of a cohesive strategic direction. A more consistent, accountable and appropriate management structure is needed to support ongoing stability and improvements.

The way PSC-led pools are created poses further potential risk. Although the PSC consults clients at the regional level, it lacks a firm commitment from clients to hire, unlike its staffing and assessment service commitments associated with signed MOUs. As a result, the need for candidates can change from the time a pool was created to the time when candidates are available, thereby creating a risk for the PSC. One good practice to mitigate potential risk is the creation of small pools of assessed candidates and assessing additional candidates when necessary.

Furthermore, the availability of PSC-led pools as communicated at the national and regional levels²¹ varies. The Operations Directorate in SASB has recently initiated a working group with a focus on developing business metrics to share good practices and create a better strategic direction for PSC-led pools. It highlights improvements and efficiencies in the management and administrative processes and shares this knowledge across the regions.

5.2.3 Processes and mechanisms

Are processes and mechanisms (including communications efforts) working as intended?

As every pool is different, the evaluation team collected the following information to help identify the procedures and mechanisms that could help lead to successful PSC-led pools. Interviews with PSC managers and clients of PSC-led pools identified certain good practices that could be considered when additional pools are developed. These include the following:

• A thorough needs analysis that factors in high-demand job classifications.
• Bilingual pools where candidates have been pre-assessed in writing and reading in second language evaluation areas.
• The level of assessment available should be aligned specifically with client needs.
• The size of PSC-led pools should be tailored to meet the regional demand for assessed candidates.
• Regions should routinely verify candidates’ interest and be more time sensitive to client needs when making referrals.
• Communications and marketing efforts should be accelerated in order to improve hiring managers’ awareness of PSC-led pools.

The evaluation also attempted to develop a preliminary picture of the characteristics of a successful pool. One important measure of success was to recover all the associated counselling and assessment costs. The evaluation used expired pools as a basis for measuring success thus far as they are no longer active and would not be generating any new revenue. Analysis of successful expired PSC-led pools (i.e. broke even or made a profit) revealed several tentative characteristics. It should be noted that these characteristics, differed depending on whether the pool was regional or NCR-based:

Regional

• Administrative level positions;
• Low number of candidates in pools (less than 30);
• Less than $15 500 in total assessment costs; and
• A comparison of regionally based PSC-led pools across levels of assessment showed that partially assessed pools tend to be more cost- effective, but as indicated through the surveys hiring managers prefer candidates that are fully assessed for language. The most successful²² region to date in terms of overall cost recovery is the Atlantic Region, with revenues of $25 545.

National Capital Region

Although there is currently only one expired NCR pool, this pool tentatively suggests the following characteristics for success:

• Focus on administrative level positions; and
• Larger number of candidates to draw from.

The table below details all PSC-led pools that are no longer available in 2009-2010 by their assessment type and their associated revenue.

This table provides costing and revenue information for PSC-led pools that are expired (i.e. candidates are no longer available for referral) and pools that are available.

To what extent are the staffing needs of federal organizations being appropriately targeted?

As mentioned earlier, interviews suggest a consistent analysis of human resources needs among federal organizations is currently not being done. As a result, it would appear untapped demand has not been measured or estimated, and the actual size of the potential market is unknown.

The time needed to staff a position is an important consideration for hiring managers. According to an October 2009 study,²³ the average time to staff a position under the new PSEA was 23.5 weeks. In interviews, clients said that the time to staff from a PSC-led pool varied considerably, ranging from two to three weeks to 24 weeks, but was generally around six weeks. Factors that influence time to staff from a PSC-led pool were pool availability, location of candidates, language skills and the candidates' level of interest.

While building pools and selling candidates in the same fiscal year would facilitate budgeting, hiring patterns need to be taken into consideration. As shown in the following chart, recruitment of indeterminate and specified terms over three months²⁴ is highest at the beginning of the fiscal year. Therefore, in order to have candidates available at the beginning of the fiscal year, pools would need to be established the previous year.

Recruitment of indeterminate and terms over three months

Recruitment of indeterminate and terms over three months- long description

It should be noted that additional time was also required as hiring managers conducted their individual assessments. The evaluation was unable to determine the total time required to staff, including the time needed by the client, as pool management did not gather this information.

Recommendation 2: Uniform processes

The PSC should develop uniform service delivery processes and standards to address service-related issues promptly and effectively. The PSC should centrally monitor satisfaction levels among candidates to PSC-led pools, as well as other relevant performance measures, such as the length of time from client requests to the delivery of referrals, to identify areas for improvement. The PSC should also implement formalized goals and objectives, operational guidelines and a risk-mitigation strategy to enable PSC-led pools to reach their full potential.

Recommendation 3: Clarification of roles and responsibilities

The PSC should identify a lead and clarify roles and responsibilities related to PSC-led pools, while maintaining their flexible nature.

5.3 Cost-effectiveness of Public Service Commission-led pools

5.3.1 Cost recovery results

Are PSC-led pools cost-effective? Are some pools more cost-effective than others and why?

Pricing for PSC-led pools is based on approved per diem rates for professional services and administration, which include direct and indirect costs.²⁵ Resources track their time based on a list of activities at either the professional or administrative rate, depending on the activity. Total costs include labour costs, which are arrived at by multiplying the number of hours for cost recovery activities by the per diem rate, as well as the cost of any Personnel Psychology Centre tests and non-salary costs, such as travel or room rental.²⁶

Two measures of cost-effectiveness were considered in this evaluation: 1) the extent to which the costs of PSC-led pools are recovered; and 2) comparative costs to clients for equivalent service from other suppliers (i.e. if fees charged to them are equal to or less than fees that other suppliers charge for similar services).²⁷

Extent to which the costs of PSC-led pools are recovered

The Integrated Services Division (ISD) and regional offices provided partial data on costs in PSC-led pools; the regions provided other additional financial data. From April 2008, when cost recovery for PSC-led pools began, to January 2010, the PSC recovered $570 300 of the $789 417.²⁸ This works out to a cost recovery rate of 72% to date.

Cash flows broken down by fiscal year demonstrate that in the first full year of operations (2008-2009) $229 800 was recovered of the $640 600 expended (i.e. a recovery rate of 36%). In the second year $340 500 was recovered and only $148 817 was expended (i.e. a recovery rate of 229%). This high recovery rate and low expense rate for the second year can be attributed partially to the fact the development costs for many of the pools were expensed in the previous year and revenue is only being realized in the second year. It is important to note that the PSC is acting within its cost recovery authority. Pursuant to the Financial Administration Act, the PSC has authority to expend revenues received in a fiscal year to offset expenditures incurred in that fiscal year. The PSC's interpretation is that total expenditures can be offset by total revenues and not revenues and expenses by service or product line. This interpretation also complies with the financial reporting requirements of the Auditor General of Canada.

If current trends continue into the third year of the initiative, there is the possibility that PSC-led pools will break even overall and potentially make a profit. It must be kept in mind that if the PSC continues to offer "high-risk" pools (i.e. with incomplete needs assessment), successful pools will have to be more profitable to cover for those that are not.

This trend of uncertainty associated with the timing of potential profits is an inherent characteristic in the type of staffing service that has no formal financial commitment (i.e. MOU or service level agreement) or an assured market. Interviews suggested that management undertook this risk with the understanding that the potential for breakeven and profit would be eventually realized.

In addition, evaluation interviews suggest that experience in developing and administering pools also contributed to the reduction of the costs of new pools. In the first year, without a clear understanding of the actual need for pre-assessed candidates, a substantial amount of money was initially invested into creating pools. However, in the second year, regions seemed to be more conservative in creating pools and continued to refer candidates from previously established pools. Therefore, PSC-led pools and inventories can be considered an asset as they may generate revenues over more than one fiscal year.

The table below presents a sampling of candidate and cost information from a variety of PSC-led pools in every region (sample used to provide overview of all pools). On average, pools of partially assessed candidates tend to be larger than those of fully assessed candidates. The most successful PSC-led pool was the CR-2/3/4/5 pool in the Atlantic Region. It had the lowest cost per candidate as well as the highest profit margin of all PSC-led pools.

* Fully assessed candidates generally cost $2 500 when selected for appointment. For partially assessed pools, the rate is $1 800 for five referrals. In these cases, if no additional request is made due to unsuccessful processes, it is assumed that each set of referrals generates one appointment.

** Estimate based on number of candidates referred.

Regional data in the previous table demonstrate that although costs and the number of candidates in PSC-led pools vary significantly across regions, some regions have had more success²⁹ with larger pools than others. For example, the NCR, where roughly 40% of the public service is located, has experienced a net gain of $21 334 from its pools to date. For fully assessed pools, the Alberta and Northwest Territories Region had success with smaller pools, with a net gain of $12 141. Two new pools have been recently established in Quebec with a total cost of $81 097 from which no referrals have been made as yet. The overall appointment rate is 7%, but smaller pools have much more effective appointment rates, which can be as high as 80%.

Key observations on costing:

• Total pool costs – Interviews and discussions with PSC management indicate that the cost of pools is not completely representative of total costs. Based on authorities from the Treasury Board Secretariat,³⁰ the PSC can recover only counselling and assessment costs. Good practice suggests there should be consistent allocation of appropriate direct and indirect costs, including program support expenses and corporate overhead to provide an accurate account of what a pool costs the PSC even though only counselling and assessment costs are recoverable. A better understanding and more complete idea of all costs, including non-counselling and assessment costs, would give management more information on which to base decisions. It should be noted, however, that the per diem rate is based both on direct and indirect costs.
• Costing and pricing issues – Interviews suggest that fees have been fixed by ISD based on their research. Following an approval process, SASB publishes a price list on the PSC Internet site and regional offices are not allowed to adjust prices. The current rigid fee structure of PSC-led pools may be one factor that limits cost-effectiveness. Pools that cover large geographic areas with multiple testing centres are expensive to administer. Regional offices could potentially meet their individual needs better with increased price flexibility.

Comparative costs to clients for equivalent service from other suppliers

To establish cost effectiveness, it is necessary to establish a cost comparator. It is, however, often difficult to establish comparators as in many cases significant differences exist between potential competitors. Establishing useful comparators for the PSC pools has been a challenge, but some useful inferences can be made.

Several case studies of collective staffing by federal organizations for their own positions were conducted to compare against the costs of PSC-led pools. Given that the majority of departmental processes reported only non-salary costs, such as PPC tests, temporary help and consulting services, a number of assumptions were made in order to estimate the actual time departmental employees devoted to these processes. Results show that collective staffing costs vary from as low as $289 to as high as $1 622 for each assessed candidate, with an average cost of around $400. The PSC's average cost for each qualified candidate is around $200. In both cases, there is a mix of fully and partially assessed candidates. This measure needs to be interpreted with caution as the more candidates there are in the pool, the lower is the unit cost. Furthermore, the average cost does not provide any insight into the real quality of candidates.

A more revealing cost-effectiveness measure is the cost per hire, which amounts to $2 530 for the PSC-led pools and to $2 017 for departmental pools. However, a number of activities have not been taken into consideration for the departmental processes, including the communication of results to candidates and the management of assessment logistics, which would add to the process cost. In addition, seven processes out of 17 were internal and these are usually less expensive to run given the smaller number of applicants. The PSC currently charges $1 800 per set of five partially assessed referrals and generally $2 500 per fully qualified candidate selected for appointment, for an overall average of $2 150. While some departmental processes are less expensive, they use internal resources that could be dedicated to more strategic functions, such as planning and core activities. The full case studies are attached as Appendix D.

The following chart demonstrates that when compared to individual departmental processes, the PSC's initiative has a good average in terms of cost per hire. In addition to their competitive costs, PSC-led pools have the potential to provide managers with easy just-in-time staffing provided that pools are established in a timely fashion or replenished on an ongoing basis.

Cost per hire by departmental process

Cost per hire by departmental process - long description

Factors identified in the case studies that contribute to lower cost per hire include:

• Partial assessment. However, in some cases they were still expensive processes as level of assessment may vary (e.g. bilingual vs. non-bilingual pools);
• EE groups targeted, which reduced the number of candidates to assess;
• Significant dropout rate at each step of the process when hurdle approach is used; and
• Smaller pools generate high appointment rates.

Another comparison was made with recruitment firms. Their fee for managing the entire recruitment process to fill administrative positions varies between 15% and 20% of the selected candidates' salary. Knowing that the average number of hires per process in the case studies was 17, the overall cost for a similar process would amount to approximately $150 000 or a cost per hire of $8 824. This is 3.5 times more expensive than PSC-led pools.

Despite these differences, some findings from the case studies could be applied to PSC-led pools:

• The more specialized the position, the more expensive it was to administer the process. These processes require tailored assessment strategies, whereas those to fill administrative positions may rely on a more generic approach.
• Larger pools seem less effective than smaller ones, because the process is less efficient when the number of candidates increases. These pools appear to yield lower appointment rates, thus increasing the cost per hire.
• Data are often skewed because the actual time and cost of salaried staff represented as sunk costs were often not included when costs were calculated. Therefore, conservative assumptions needed to be made.

Recommendation 4: Costing review

SASB and particularly ISD should consider conducting a costing review to define a consistent and clear methodology and reporting process for all regions. This cost-reporting process should ensure that total costs related to all PSC-led pools are accurate and comparable among regions.

5.4 Alternative business models

Minimizing financial risk: Establishing service requests and memoranda of understanding

Most activity in SASB is linked to individual staffing processes and pools. These activities present low financial risk to the PSC because they are implemented only with agreements that ensure costs are covered. When clients bring a staffing request to SASB, they have two options:

• A memorandum of understanding (required for individual process staffing and pools above $50 000) or a service request (required for those below $50 000); or
• For a fee, hire a candidate who has already been screened in, assessed and placed into a pool.

Mitigating financial risk: Setting an acceptable range for cost recovery

The Government of Canada Guide to Costing discusses cost recovery in the following way:

Cost recovery refers to setting charges to cover some or all of the costs incurred in providing a product or service, rather than funding the product or service solely out of general tax revenues. The recovery of less than full cost may be justified on the basis of policy, program, or administrative grounds. Nevertheless, it is important to be aware of the gap between the costs of delivering a service and the revenues that will be generated.³¹

The PSC could apply the concept of "normal efficiency" to the administration of pools in all regions. An efficient business function is one that is handled in a "normal" manner – neither better nor worse than what may be reasonably expected. This approach may be useful because it is flexible enough to suit varying guidelines for delivery that are outside the control of the service deliverer, while still focusing on efficiency. Also, this approach may lessen the temptation to maintain and nurture a culture of detailed comparative costing among multiple providers who focus on the lowest cost. Rather, it encourages providers to emphasize the quality of the service within acceptable norms.

Exploring business partnerships to increase the efficiency and effectiveness of PSC-led pools

In interviews, pool managers had several suggestions for making PSC-led pools more efficient and effective by developing business partnerships with other federal organizations. PSC management would be required to establish, co-ordinate and implement these partnerships. This would permit the PSC to speak with one voice to external partners and achieve positive outcomes. Two areas, as detailed below, could benefit from further exploration:

• Develop common assessment processes and tools for certain positions to simplify the process. A study³² found a high level of common merit criteria existed in SoMCs across federal organizations. These SoMC are used for recruitment, both within the same groups and levels, and between different groups and levels, for high-volume, entry-level positions; and
• Partner with Service Canada,³³ for example, for some PSC-led pools services.³⁴ Combining logistics can maximize the efficiency and effectiveness of PSC-led pools, especially in delivering recruitment and testing services across large geographic areas.

6 Summary of recommendations and management response

Recommendation 1: Clients' needs and outreach

The PSC should anticipate and map out clients' needs through ongoing communications and environmental scanning and factor in high-demand regions and high-volume job classifications (AS and CR streams). Particular attention could be paid to niche segments. Efforts should be made to target small federal organizations, positions in remote locations and those regions without a human resources presence, where needs are potentially the greatest. PSC-led pools should promote their services more widely to increase market penetration.

Outreach mechanisms should reach stakeholders to increase awareness of PSC-led pool activities and their contribution to the hiring process, such as reducing the length of the hiring process and communication of applicant status in a pool, in an effort to increase potential uptake.

Management response:

The management team of the Staffing and Assessment Services Branch (SASB) agrees with this recommendation and will respond by:

• Conducting a marketing analysis, which includes the determination of client needs and environmental impacts (internal and external);
• Developing marketing techniques and tools to respond to client needs and better promote our services; and
• Ensuring the Operations Directorate (OD) and the National Client Services Directorate (NCSD) work closely with the Integrated Services Division (ISD) to develop and implement consistent outreach and communications methods.

Recommendation 2: Uniform processes

The PSC should develop uniform service delivery processes and standards to address service-related issues promptly and effectively. The PSC should centrally monitor PSC-led pool satisfaction levels among clients, as well as other relevant performance measures, such as the length of time from client requests to the delivery of referrals to identify areas for improvement. The PSC should also implement formalized goals and objectives, operational guidelines and a risk-mitigation strategy to enable PSC-led pools to reach their full potential.

Management response:

SASB's management team supports this recommendation and has responded by:

• Tasking the Pool Management Working Group, via SASB Operations Committee, to analyze current regional procedures and develop uniform service delivery objectives, processes and tools. Regional flexibilities will be considered and apparent and perceived risks will be incorporated throughout the decision making process; and
• Having NCSD, OD and ISD continue their partnership in redesigning the current client satisfaction survey to respond to SASB and client needs.

Recommendation 3: Clarification of roles and responsibilities

The PSC should identify a lead and clarify roles and responsibilities related to PSC-led pools, while maintaining the flexible nature of pools.

Management response:

SASB supports this recommendation by agreeing to the following:

• The Vice-President of SASB is to consider all viable options when determining who will be responsible for the overall direction of PSC-led pools and inventories.
• The Vice-President will report updates on SASB's programs to the Executive Management Committee.

Recommendation 4: Costing review

SASB and particularly ISD should consider conducting a costing review to define a consistent and clear methodology and reporting process for all regions. This cost-reporting process should ensure that total costs related to all PSC-led pools are accurate and comparable among regions.

Management response:

The management team of SASB has already committed to a costing review and is therefore supportive of this recommendation. Also, ISD will review and implement effective standardized cost recovery tools and procedures and ensure consistent use of these in all service delivery offices.

7 Management action plan

8 Limitations of the evaluation

• Based on limited research into time to staff from PSC-led pools, the evaluation team could not generalize that PSC-led pools had reduced time to staff across government. However, in many instances, the time needed to staff a position was reduced.
• PSC-led pools are a new PSC service. For this reason, a thorough study of the performance of PSC-led pools over time could not be done.
• Some of the PSC-led pools reviewed for the evaluation were set up recently. Because no referrals had yet been made, it was impossible for the evaluation team to study the performance of these pools.
• A targeted number of interviews with clients were conducted. Responses were received from 519 (from a sample size of approximately 4 000) hiring managers. As a result, it was difficult for the evaluation team to widely generalize that PSC-led pools are in fact a time-saving and cost-effective option for hiring managers.

Appendix A: Client and non-client organizations interviewed

• Public Works and Government Services Canada
• Correctional Service Canada
• Defence Research and Development Canada
• Canada Revenue Agency
• Citizenship and Immigration Canada
• Public Prosecution Service of Canada
• Service Canada
• Canada Border Services Agency
• Health Canada
• Environment Canada
• Canadian Heritage
• Natural Resources Canada
• Department of Finance Canada

Appendix B: PSC-led pools, January 2010

Appendix C: Public Service Commission-led pools logic model

Public Service Commission-led pools logic model

Public Service Commission-led pools logic model - long description

Appendix D: Comparative analysis – Collective staffing case studies

The following case studies are based on collective processes run by four federal organizations. The level of detail varied from one organization to the other and costs were not consistently captured. Most of them included only non-salary costs, such as PPC tests, consulting services and temporary help. In order to make a fair comparison, expenses were adjusted to reflect the potential cost of each activity based on the number of candidates at each step of the process. The following conservative assumptions were made further to consultation with finance and HR specialists:

• The average public service salary, set at $66 629³⁶ , was used for all recruitment activities except for test supervision, in which case a CR-04 salary was used.
• Two hours was allocated for the drafting or update of the SoMC given that qualifications for administrative positions are often similar from one intra-departmental process to the other.
• As determined in the PSRS evaluation, manual screening on experience takes on average 15 minutes for each candidate.
• Consulted organizations tend to reuse in-house tests for administrative positions. Therefore, two hours was allocated to the development or update of tests and another two hours to develop or update the rating guide.
• Room rental costs were only included if reported.
• Test supervision was estimated at two hours per test with a maximum of 20 candidates per group.
• Time to mark in-house tests was estimated at 20 minutes for each candidate.
• Interview time was set at 60 minutes per candidate, including a 15-minute scoring period.
• A 30-minute period per candidate was allocated to reference checks.
• Pool administration time was excluded from the analysis.
• Employee Benefit Plan and 20% overhead were included.

Case A: Public Service Commission

The Public Service Commission held three collective staffing actions from July 2008 to August 2009. Two processes had four pools aimed at staffing administrative and clerical positions (CR-4/5) with varying tenures and locations.

The clerical positions had two CR-4 pools and two CR-5 pools. During the screening and assessment phases, there was no segregation, as all levels were assessed using the same criteria. The assessment consisted of three tests (GCT1-207, OST-201 and OST-203), interviews and reference checks. Of the 296 candidates invited to the assessment, 52 were essentially qualified. Fifteen of them have been appointed to positions to date. The total costs for cost recovery were approximately $38 781, which resulted in a cost of $746 per qualified candidate and $2 585 per hire.

Case B: Agriculture and Agri-Food Canada

Agriculture and Agri-Food Canada (AAFC) has created a wide range of collective staffing processes. These processes covered administrative (AS, CR), various levels of policy analysts (ES) and computer systems/analyst positions. The collective staffing actions resulted in the appointment of 158 candidates. Several other pools are in progress; statistics for them are unavailable.

The first pool was created in May 2007. The AS-3 administrative assistant pool had 151 candidates apply, of whom 44 were fully qualified and 21 were eventually appointed to positions. The total cost was estimated at approximately $22 518, or $512 per fully qualified candidate and $1 072 per hire.

A second pool, created in June through August 2007, was to establish a pool of partially qualified candidates for administrative positions at the CR-4, CR-5, AS-1 and AS-2 levels. A total of 4 877 candidates applied for the various levels, with 551 being partially assessed. Ten candidates were appointed to the CR-4 level, one to CR-5, and a total of 40 candidates to the AS-1/2 levels. The estimated total cost to run the process, which included temporary help to administer exams, consultants for screening purposes, use of internal employees and room rental, was $158 659. The approximate cost per partially assessed candidate was $288 and $3 111 per hire.

In April 2008, an ES-5 senior trade policy analyst pool was created. Of the 243 candidates who applied, 19 candidates qualified for the essentially qualified pool. Of those who qualified, five were appointed to AAFC. With consultants marking the exams and conducting reference checks, and temporary help administering exams, the estimated total cost was $30 824, or $1 622 per essentially qualified candidate and $6 165 per hire.

An AS-2 administrative pool was created in March 2008. Of the 419 candidates who applied, 46 were partially assessed. Of those, 28 have been appointed to positions. The estimated total cost incurred in creating this pool was $13 276, or $289 per partially assessed candidate, with a low cost per hire of $474.

In May 2008, AAFC created an ES-6 senior policy analyst essentially qualified pool. There were 132 applicants. Fifteen of them were essentially qualified for the pool, at a cost of $1 475 per candidate, for an estimated total of $22 127. Since its creation, six candidates from the pool have been appointed to various positions for a cost per hire of $3 688.

Following up on the AS-1 pool created in June 2007, a subsequent pool was created in August 2008. Of the 370 candidates who applied, 63 were fully qualified in the pool and 10 were appointed. The estimated cost to create the pool was $22 260, at an approximate cost of $353 per fully qualified candidate and $2 226 per hire.

A computer system pool (CS-3) was created in September 2008. Of the 609 candidates who applied, 71 were fully qualified for the pool. The estimated total cost for this initiative was $37 013, or $521 per qualified candidate. Eighteen appointments have been made from this pool, with a cost per hire of $2 056.

It is important to note that AAFC demonstrated an interest in using PSC-led pools in 2010-2011 to fill administrative positions rather than setting up its own pools. If it is found to be cost-effective, the department may rely on the PSC for this type of position.

Case C: Transport Canada

Since 2007, Transport Canada (TC) has initiated five collective staffing actions. These pools were created to fill potential voids in administrative support (CR), administrative assistant (CR-5 and AS-1), aircraft certification (ENG) and civil aviation safety (TI). Tenure details for each of the pools range from term to indeterminate positions and have various language requirements. These staffing pools do not operate on a cost recovery basis.

The CR-4 pool for administrative support clerks was assessed using one PPC test (GSPAT-120), a general written exam, an interview and reference checks. Of the initial 953 applicants, 41 were fully assessed and 13 were eventually appointed to positions. The CR-5 and AS-1 pool was completed at the same time as the previous pool. A similar approach was used to assess the candidates: two PPC tests (GCT2-314 and WCPT-351), an interview and reference checks. Ninety-six candidates took part in this process, and 30 were successful and fully assessed. The estimated total cost for pool processes, including temporary help and consultants, was $29 991. The approximate cost per qualified candidate was $441 and $909 per hire.

Another pool was created with the intent to staff civil aviation safety inspector, maintenance and manufacturing (TI-6) positions. The aim is to staff indeterminate positions in various locations. Candidates were fully assessed using a PPC Test (WCPT-351), exam, interviews and reference checks. Of the 651 applicants, 138 candidates were invited to participate in the assessment process, and 27 were found qualified. Eight have been appointed to positions to date. The estimated total cost to run the process was $11 555, costing approximately $428 per fully assessed candidate and $1 444 per hire.

An external pool was created for aircraft certification engineers (EN-ENG-4). This process is in the referral phase and is intended to staff indeterminate positions in various locations. The candidates were essentially assessed using a knowledge exam, interviews and reference checks. Of the 127 candidates invited to participate, 16 were essentially assessed, with three candidates appointed to positions to date. The estimated cost has not been finalized, so we are unable to determine the cost per candidate assessed. The estimated cost to run the process was $13 477, costing approximately $842 per fully assessed candidate and $4 492 per hire.

Case D: Human Resources and Skills Development Canada

Since 2007, Human Resources and Skills Development Canada (HRSDC) has initiated three collective staffing actions. These pools were created to fill ES-2/3 and PM-2 positions. These staffing pools do not operate on a cost recovery basis.

These processes were launched to create fully assessed pools and they did not make use of standardized tests. Of the initial 3 050 applications received, 1 700 were screened in and the assessment phase resulted in 223 appointments. The total cost for the three collective processes was $1 302 075, resulting in a cost per hire of $5 838. The following table provides overall departmental averages, excluding HRSDC processes given their large size and costs, which would have skewed the results. A separate table on HRSDC processes is provided.

Findings

1. Full costing information related to these pools is not available because sunk costs (i.e. employee per diem salaries) were not included in the total cost of establishing the pools. Therefore, assumptions were made to estimate these costs.
2. The more specialized the requests, the more expensive the process is to administer as there may be several qualifications to assess requiring a tailored assessment strategy. For administrative positions, organizations tend to use a more generic approach that does not require the hiring manager to be involved in the assessment process, hence minimizing costs and time to staff.
3. Larger pools seem to be less effective than smaller ones. The collective staffing process loses its efficiency as the number of candidates increases. These pools tend to yield lower appointment rates, thus increasing the cost per hire.

1 Relevance: the extent to which a program addresses a demonstrable need, is appropriate to the federal government, and is responsive to the needs of Canadians. Source: Treasury Board of Canada Secretariat, Policy on Evaluation, 2009. (return)

2 Performance:  the extent to which effectiveness, efficiency and economy are achieved by a program. Source: Treasury Board of Canada Secretariat, Policy on Evaluation, 2009. (return)

3 For more detailed information, see: Staffing and Assessment Services and Pools Available (return)

4 PSC Web site. Staffing and Assessment Services (return)

5 PSC Annual Report, 2008-2009, p. 131. (return)

6 The total number of candidates in all PSC-led pools is higher than 21 745 because job seekers often apply to several pools at the same time. (return)

7 It is unclear how many referrals resulted in hires. Insufficient data exist on the total number of appointments. 1 000 candidates have been referred of 4 592 assessed or 22%. (return)

8 This figure is based on pools information collected from regions in January 2010. (return)

9 PSC Web site: Staffing and Assessment Services (return)

10 PSC Web site  Mission, Vision and Values Statement (return)

11 TBS Web site. 2008-2009 Public Service Renewal Action Plan (return)

12 Treasury Board of Canada Secretariat Common Services Policy 2006 (return)

13 Justice Canada Web site. Public Service Employment Act (return)

14 Public Service Commission Annual Report 2008-2009 (return)

15 Public Service Commission, 2010-2011 Draft Report on Plans and Priorities (return)

16 Summary Report of the Pre-Qualified Pool Evaluation (return)

17 Ibid. (return)

18 Sixteenth Annual Report to the Prime Minister on the Public Service of Canada, March 31, 2009. (return)

19 Earlier reports placed the same priority on the time to staff. As early as 2001, the Auditor General of Canada stated that “hiring managers need more central and departmental inventories of pre-qualified individuals that they can draw from quickly to meet their recruitment needs”. (return)

20 Fisheries and Oceans Canada, Human Resources and Skills Development Canada, Department of National Defence, Public Works and Government Services Canada, Transport Canada, Veterans Affairs Canada and Privy Council Office. (return)

21 Data are also gathered as to anticipate departmental needs from Human Resources Council, National Staffing Council, Personnel Advisory Group, APEX and PSC DM Advisory Committee. (return)

22 This region was successful in terms of cost recovery, but success can also be defined by quality of referrals, which is dependent on assessment tools used, and client satisfaction. (return)

23 Time to Staff in the Federal Public Service — An Update: A Summary Report by the Public Service Commission of Canada, October 2009. (return)

24 Note: Data for 2009 - Q4 and possibly 2009 - Q3 is under estimated until all the 2009-2010 appointment data are available at the end of April , 2010. The FY totals will not match the PSC Annual Report data for recruits because the appointment effective date was used instead of the ROST received date. (return)

25 Indirect costs include program support expenses and corporate overhead. (return)

26 On an appointment basis, the client pays a fee for each appointment while on a referral basis, the client pays for a referral (partially assessed only) regardless of whether they are appointed. (return)

27 Private industry guidelines to staff a position with an annual salary of $40 000 costs approximately $3 300 for a fully assessed candidate according to the Council of Human Resources Professionals. (return)

28 $789 417 includes assessment costs and other related costs, such as room rental, operations and maintenance and per diem salaries. $570 300 has been recovered through fees to client organizations for candidates. Some of the pools' costs included in $789 417 are new pools from which referrals have not been made yet. When these pools begin making referrals, the amount of costs recovered is expected to increase. (return)

29 These regions were successful in terms of cost recovery, but success can also be defined by quality of referrals, which is dependent on assessment tools used, and client satisfaction. (return)

30 Treasury Board Submission. Subject: The Public Service Commission's vote netting authority. (return)

31 See TBS Guide to Costing (section 7) (return)

32 PowerPoint Deck: Hiring Optimization, PSSMP Services and Products Advisory Group, September 15, 2009. (return)

33 Service Canada has 600 service centres across the country. (return)

34 The PSC has only seven regional offices across the country. (return)

35 PSC Website. PSC-led pools and inventories. (return)

36 Source: Compensation and Labour Relations, Treasury Board Secretariat (return)