Evaluation of the Public Service Commission's Post Secondary Recruitment Program

Final Report
May 10, 2010

Table of Contents

• Executive Summary
• 1. Introduction
• 2. Methodology
  • 2.1 Evaluation Objectives and Purpose
• 3. Post Secondary Recruitment Profile
  • 3.1 Background
  • 3.2 Program Governance
  • 3.3 Roles and Responsibilities
  • 3.4 Program Resources
• 4. Logic Model
• 5. Findings and Analysis
  • 5.1 Rationale and Relevance
  • 5.2 Design and Delivery
  • 5.3 Outcomes/Success
  • 5.4 Cost-Effectiveness and Alternatives
• 6. Recommendations
• 7. Evaluation Limitations
• 8. Management Response and Action Plan
  
• Appendix A: Logic Model/Intended Outcomes
• Appendix B: Evaluation Matrix
• Appendix C: Perceptions of the Federal Public Service
• Appendix D: Importance of Various Job Aspects
• Appendix E: Values
• Appendix F: Self-Rated Confidence and Perceptions of Risk-Taking
• Appendix G: International Comparisons

Executive Summary

What we examined

This evaluation was carried out for the Public Service Commission's (PSC) President and Executive Management Committee in accordance with the Risk-based five year Evaluation Plan 2009-2014. This report presents the findings and recommendations of the Evaluation of the Post Secondary Recruitment (PSR) Program.

The PSR Program was created in 1973 to help federal organizations address their need to fill entry-level officer positions. The Program was evaluated for the Corporate Management Practices and Evaluation Directorate of the PSC and included multiple lines of evidence:

• A review of program documentation, relevant literature and data related to the Program;
• Survey of "potential recruits" to the federal public service;
• Survey of recent hires to the federal government through the PSR;
• Survey of recent hires to the federal government through non-PSR methods;
• Survey of federal hiring managers; and
• Interviews with key personnel.

Why it is important

For over 100 years, the mandate of the PSC has been to ensure that the appointment of public service employees is free of political patronage, is fair and is based on merit. In recent years, efforts have been made to modernize recruitment functions. The Privy Council of Canada has identified modernized recruitment as a priority, with the goals of ensuring that a) the public service has the people and the skills it needs for the future; b) each department and agency is equipped to meet its needs for new personnel and c) recruitment processes are accessible, credible and efficient. The National Area of Selection (NAoS) policy is another driver of change, requiring that a national area of selection - rather than specific geographic areas - be established for all external advertised appointment processes.

At the same time, Canada's public service is facing unprecedented challenges in its renewal efforts - it is aging faster than the Canadian population, it needs to better reflect Canada's diversity, technological influences continue to change the nature of work, it operates in an extremely competitive labour market and an environment of rising public demand for quality services. In addition, the Clerk of the Privy Council set a goal of hiring 3 000 post-secondary graduates for 2007-2008 and then increased that target to 4,000 graduates for both 2008-2009 and 2009-2010. The PSR Program is one element supporting federal organizations to meet public service renewal targets for the hiring of new or recent graduates into entry-level officer positions.

What we learned

Relevance

The PSR Program fits into the PSC's broader mandate of supporting staffing in the public service. Its use has increased since the introduction of the Clerk's targets from 453 hires in 2006-2007 to 1,697 in 2008-2009. Despite this, the PSR Program continues to be an under-utilized resource with only about one-third of the hiring managers surveyed having used it. Its planned future use is expected to be equally low.

Performance: Design and Delivery

Limited awareness and knowledge of the PSR Program are important factors contributing to low utilization. There appears to be some lack of understanding of what the PSR Program is, its purpose and the needs it can address. However, other factors may be affecting usage. First, the interests and preferences of hiring managers when recruiting for entry-level officer positions are perceived to differ from the design of the PSR Program:

• Hiring managers not having used the Program place little importance on test results in entry-level recruitment, emphasizing other factors such as personal suitability. A majority of those having used the Program feel that the tests are an accurate assessment of applicants.
• Many hiring managers look for experience, even at the entry level.
• Hiring managers placed importance on having flexibility and control in the process. There is a perception that they may not be able to obtain this through the PSR Program.

In looking at communication from the applicant perspective, we found awareness to be low among potential recruits, though the number of applications has grown steadily over the last three years to over 95 000 in the Fall 2009 campaign. Additionally, university representatives noted significant variation in the promotion and communication of the Program across Canada. PSC management noted that the volume of applications is so high that further focus on communication is seen as unnecessary.

Hiring managers also expressed an appetite for additional options or approaches not currently available within the design of the PSR Program:

• Over half the hiring managers surveyed would be more apt to use the PSR Program if language testing were available, the need being greatest in the National Capital Region (NCR) and the Québec region.
• Close to half would be more apt to use the PSR Program if the availability of mid-career professionals was increased (reflecting their interest in candidates with experience).
• A similar number expressed an interest in year-round intake and testing.

PSC managers express concern with the current approach to testing all interested applicants: a large number are tested annually and only a small proportion is ever referred. This is seen to reduce the efficiency and effectiveness of the Program for both the PSC and applicants. However, testing is perceived to play an important role in volume management, providing an initial screening method for applicants prior to referral. Existing tests are rated fairly positively by hiring managers who have used the Program, but less well by applicants (both groups rate the Written Communication Proficiency Test (WCPT) higher than the Situational Judgement Test (SJT) and the Graduate Recruitment Test (GRT)).

Despite the concerns raised regarding design, it is important to note that a large number of those hiring managers who actually have used the Program were satisfied with the results.

Performance: Success

Evaluation findings reveal that use of the PSR Program, while increasing, remains moderate. This is particularly evident in the regions where less than one-third of the nearly 1,700 appointments occur. Furthermore, it does not appear that the PSR Program is consistently proposed as a recruitment tool by human resources (HR) advisors, nor used by managers when a recruitment process for an entry-level officer position is initiated. Hiring managers who have used the PSR Program to recruit are largely satisfied with the quality of candidates as well as with the number of candidates and referrals.

Factors that are believed to impede use of the Program are the cost and inconvenience of national recruitment, suggesting that there is some disconnect between the actions of hiring managers and policy directives. Despite the requirement that NAoS be met for all external recruitment, hiring managers interviewed appear more concerned with factors such as cost, timing and convenience.

In 2008-2009¹, federal organizations reported that more than 5,159 post-secondary graduates had been appointed or offered appointments, of which 1,697 were appointed as a direct result of the PSR Program. Approximately 2,337 post-secondary graduates were hired through other means giving an idea of the potential market for PSR services.

Over half the hiring managers surveyed agree that the PSR Program helps federal organizations to meet public service renewal targets. They also agree that candidates hired through the PSR Program have generally performed well in positions for which they were recruited, and that these candidates have generally been promoted within the public service. Additionally, those who have used the Program are satisfied with the speed and number of referrals received (both from the General Inventory and tested career choices) and with their overall quality.

The PSR Program is seen as providing post-secondary students with an avenue of access to jobs in the public service. Most potential and recent recruits list it first as the method they used to apply to the public service and most recent recruits through the PSR Program believe it provides university graduates with access to public service job opportunities.

Interview respondents agree that the PSR Program is designed to contribute to a competitive and representative public service, through recruitment based on fairness, access and transparency. Furthermore, the PSR Program respects the requirement that positions be advertised nationally under the NAoS. The demographic profile of PSR applicants has remained stable over the past three years. The proportion of visible minority applicants within the PSR Program was higher than workforce availability but for Aboriginal candidates and persons with disabilities the proportion of applicants was lower.

Performance: Cost-Effectiveness and Alternatives

Findings indicated that the PSR Program is not used consistently as the primary central mechanism to recruit new entry-level officers. In fact, hiring managers more often report using student bridging and general recruitment than the PSR Program. From 2007-2008 to 2008-2009, 713 more appointments were made through the PSR Program for a 72% annual increase, while Federal Student Work Experience Program (FSWEP) new hires stayed relatively steady at 22% of the total, despite the increase in absolute numbers of 236 new hires for a 26% year over year increase. The data suggests that while the PSR Program is making inroads into the potential market, in 2008-2009 there was still room to increase market share by 2 337 new hires or 45%. Yet over half of hiring managers believe that non-PSR approaches better meet their hiring needs than does the PSR Program.

The limited data available on the Program costs and costs of alternative recruitment approaches restrict the cost effectiveness analysis that can be performed. The PSR Program is not currently part of the PSC cost recovery regime. However, the fact that services are provided for free is an important factor and interview respondents are not certain they would continue to use the Program if cost recovery were introduced. One improvement to cost effectiveness being explored by the PSR Program is the introduction of on-line or unproctored Internet testing, although this will only be available for one test within the next year and other tests in the next 2-3 years.

As a comparison point, the evaluation also examined several other recruitment programs in existence. While recruitment programs to the public service in Australia, the United Kingdom and the United States provide an interesting contrast, these programs are quite different in that they include a training and development component and only offer successful candidates employment following several work placements. However, some opportunity for learning from the experience of public service recruitment in the United Kingdom was identified.

Recommendations

Evaluation findings suggest that the PSR Program is generally meeting its objective of being a national recruitment program for the federal government. It fits into the PSC's broader mandate of supporting staffing in the public service and its use among hiring managers is increasing. In addition, the PSR Program is viewed quite favourably among both recent hires recruited through the Program and among hiring managers (particularly those who have used the Program in the past).

However, the evaluation does point to areas of possible improvement across three key areas: communication, outreach and marketing; testing and assessment and alignment with hiring needs.

Communication, Outreach and Marketing:

1. Increase the focus on marketing the PSR Program to both hiring managers and HR advisors within the NCR and in the regions:

• Planned future use of the Program remains low, and increased efforts to promote the Program and communicate its benefits are needed. In particular, a stronger focus on marketing of the Program and its benefits within regions is needed.
• There is a need to ensure that hiring managers are aware of the various recruitment tools available for student and / or graduate hiring. Distinctions between recruitment programs (PSR, FSWEP, CO-OP, etc.) need to be clarified. There is a lack of understanding of what the PSR Program is, the value it can offer hiring managers, when and how it can benefit them and how to use it.
• There is a need to emphasize the flexibility and control managers can have using the PSR Program and to address and overcome factors impeding its use. Given the interest in flexibility and a tailored focus, there is also a need to increase promotion of the availability of tested career choices.
• There is a need to promote that subsequent screening, language testing and other assessments are available on a cost recovery basis when accessing the general inventory.

2. Increase clarity in communication tools, for example, with respect to the steps in the process for ease of applicant use and ease of understanding by universities / colleges (Web site, outreach, presence on campus, etc.). Increase the focus on targeted marketing to potential applicants in shortage groups with a view to attracting the right applicants. A more consistent approach to the marketing of the Program across the regions is also needed to ensure broader national access. This could include:

• Seeking applicants in the fields in demand (whether at universities or colleges); and
• More consistent and targeted marketing of the Program to universities and more applications from target employment equity groups.

Testing and assessment

3. Explore the complement of tests to better control costs and increase efficiencies. This could involve:

• Ensuring that tests used for screening applicants best fit the job requirements (i.e., not all tests are relevant to all jobs). Hiring managers and applicants felt that the Written Communication Proficiency Test was the most useful indicator of candidate performance.
• Controlling costs by exploring methods to reduce the number of candidates tested.

4. Explore options for on-line or unproctored Internet testing to increase efficiency and facilitate the process for applicants.

5. To better meet hiring managers' preferences, consider Program alterations to include additional elements such as experience and language testing (on a cost recovery basis).

Greater alignment with hiring needs

6. Explore methods to better align the number and type of applicants with the needs of hiring organizations and hiring managers. These could include:

• Increasing the linkages of the PSR Program to HR planning and working in partnership with hiring federal organizations to ensure that the annual campaign is focused in areas of their needs.
• Exploring redesign options for the General Inventory to more specifically meet organizational and regional needs.
• Identifying experience as a field in the application process to enable managers to screen on this requirement.

1. Introduction

The purpose of this report is to present the findings and recommendations of the Evaluation of the Post-Secondary Recruitment (PSR) Program. The evaluation was conducted between September 2009 and January 2010. This section provides a description of the PSR Program, the purpose of the evaluation and the methodology employed.

2. Methodology

The evaluation was conducted for the Corporate Management Practices and Evaluation Directorate of the Public Service Commission (PSC). It included multiple lines of evidence:

• A review of program documentation, relevant literature and data related to the Post-Secondary Recruitment (PSR) Program;
• An on-line survey of 985 potential recruits to the federal government (i.e., university graduates less than 35 years of age who have not previously worked for the federal government);
• An on-line survey of 501 university graduates who have been recruited to the public service through methods other than the PSR Program;
• An on-line survey of 483 university graduates who have been recruited through the PSR Program;
• An on-line survey of 519 federal government hiring managers; and
• Interviews conducted with representatives of universities engaged in the PSR campaign in campuses across Canada (n=11); PSR Program and PSC management engaged in the delivery of the PSR Program (n=10); and hiring managers and human resources (HR) advisors (n=11).

2.1 Evaluation Objectives and Purpose

The purpose of this evaluation was to assess the extent to which the PSR Program is achieving its objectives and outcomes and the extent to which it is effective and provides value for money. The evaluation also asks the broader question of how well the federal government is doing in terms of recruiting talent during this current critical period of renewal and transition in order to better understand the context in which the Program exists. Appendix A is a logic model of the PSR Program.

This evaluation was preceded by the development of an evaluation framework that identified issues and questions the evaluation would address as well indicators and methods. The Evaluation Matrix is provided in Appendix B.

3. Post Secondary Recruitment Profile

3.1 Background

The Post-Secondary Recruitment (PSR) Program was created by the Public Service Commission (PSC) in 1973 to help federal organizations address their need for candidates to fill entry-level officer positions. The PSR Program targets university graduates in a multiplicity of disciplines. Over its history, the Program has worked to establish itself as the government's primary vehicle for recruiting graduates into the public service. The objectives of the PSR Program are to:²

• Attract new or recent graduates who are interested in pursuing a career with the public service with little or no experience;
• Enable federal organizations to meet recruitment challenges by accessing and hiring qualified university and college graduates for entry-level officer positions; and
• Provide an approach to recruitment that expedites collective staffing at a national level for departments, agencies and functional communities.

The Program has evolved considerably over time, with the addition of tested career choices, career fairs to promote the Program and the evolution to an on-line application process. In particular, the size and scope of the annual campaign has increased in the past four years (2006-2007 to date), making this period the primary focus of the evaluation. The priority placed on public service renewal has dramatically increased in that time, with the identification by the Clerk of the Privy Council, of targets for new entry-level hires. The PSR Program introduced a tested general inventory in the 2006-2007 campaign that is intended to further assist hiring departments and agencies in meeting the Clerk's commitments. Annual renewal targets have also helped to increase the focus on the PSR Program.

Through the PSR Program, the PSC offers both a general inventory and organization-specific job opportunities (referred to as tested and non-tested career choices) to meet the needs of hiring managers. The general inventory offers a pool of partially assessed university graduates. To be included in the general inventory, candidates have to successfully pass three standardized tests: the Situational Judgement Test, the Graduate Recruitment Test and the Written Communication Proficiency Test. Hiring departments and agencies can request referrals from this inventory on the basis of employment equity criteria, education / area of study, test results, citizenship and work location.

Organization-specific career choices provide hiring managers with the means to recruit for specific positions or those relating to a functional community. Federal organizations wishing to advertise tested career choices must submit their statement of merit criteria to the PSC before the fall campaign begins. Non-tested career choices can be posted throughout the year.

Interested candidates apply to the PSR Program on-line at jobs.gc.ca. There, they must complete a profile that contains information such as education, language and mobility and provide a copy of their résumé. If they decide to apply for a specific opportunity or the General Inventory, they must complete all the application requirements before the closing date. In the fall of each year, candidate testing occurs nationally and internationally in over 70-80 locations. If candidates wish to remain in the inventory for more than one year, they must re-apply (although test results can be brought forward, eliminating the need to write tests a second time).

It is important to note that not all candidates applying through the PSR Program are necessarily young or recent graduates. The Program was initially designed to attract recent graduates but cannot restrict other applicants. Applicants can include individuals who have returned to school to change careers and who may have significant work experience. Furthermore, in order to respect human rights concerns, the Program does not discriminate based on the recency of graduation as recency must be established as a bona fide occupational requirement.

Once the PSR Program has referred candidates to federal organizations, these organizations are responsible for conducting further assessment such as interviews and reference checks, thus assessing selected candidates based on the statement of merit criteria.

3.2 Program Governance

The PSR Program is managed by National Client Services Directorate, with support from the regions. Regions provide coordination for annual testing administration (e.g., details regarding facilities, special accommodation testing, etc.) and also provide support for promotion at career fairs at universities within their regions.

An organizational chart depicting the management of the PSR Program is provided in the following chart. The Manager for the PSR Program reports to the Director of Resourcing, who in turn reports to the Senior Director of National Client Services and then to the Director General of National Client Services. The Manager of the PSR Program has six individuals reporting to them. These positions include one Program Officer, two Program Coordinators and three human resources (HR) assistants. As well, a PSR Coordinator is located in each regional PSC office across the country.

Organizational Chart:

Director General, National Client Services
Senior Director, National Client Services
Director, Resourcing
Manager – Post Secondary Recruitment
PSR Regional Coordinators (6)
Program Officer
Casual – HR Assistant
FSWEP Student – HR assistant
Program Coordinators (2)
HR Assistant

Source: PSR Program Files
See also: Organization Chart - Graphic version

3.3 Roles and Responsibilities

The PSC has sole responsibility for the management and implementation of the PSR Program. The Program creates awareness among federal organizations and encourages use of the Program among hiring managers and the HR community. The PSR Program also conducts outreach to promote the Program and the public service as an employer to potential applicants in post-secondary institutions across the country. The PSC works in partnership with post-secondary institutions to promote the PSR Program and conducts career fairs on numerous campuses each year.

The PSR Program has the responsibility of managing the logistics associated with the intake campaign, including invigilators for testing, testing facilities (e.g., room rentals) and accommodation for candidates with special requirements (estimated to be 1.5% of applicants). Tests are provided and scored by the Personnel Psychology Centre, on the Program's behalf.

3.4 Program Resources

The PSR Program has evolved substantially over time and was largely re-designed in 2006-2007. Costs from the past three campaigns are summarized below, based on information provided from the Program.

4. Logic Model

In general terms, a logic model illustrates how the activities of a policy, program or initiative are expected to lead to the achievement of final outcomes. The logic model serves to identify the short, medium and long-term outcomes of a policy, program or initiative. In that sense it serves as a roadmap for the program. The logic model for the Post-Secondary Recruitment Program is presented visually in Exhibit 1.1, and described in detail in Appendix A.

5. Findings & Analysis

This chapter presents findings of the Evaluation of the Post-Secondary Recruitment (PSR) Program, organized by evaluation issue and question.

5.1 Rationale and Relevance

Under the issues of rationale and relevance, the evaluation explored the extent to which the Program is consistent with government-wide Public Service Renewal priorities and any needs not currently being met by the Program. Existing documentation and findings from interviews with key personnel were used to address these questions.

Question 1: To what extent is the Program consistent with government-wide Human Resources Renewal priorities (e.g., implementation of the Public Service Employment Act, National Area of Selection, Clerk of the Privy Council's priorities)? How does it complement, support or duplicate other recruitment efforts?

The PSC is mandated under the Public Service Employment Act (PSEA) to safeguard the integrity of the public service staffing system and the political neutrality of the public service.³ Under the Act, the PSC works closely with government but is independent from ministerial direction and is directly accountable to Parliament. The PSC's strategic outcome is to provide Canadians with a highly competent, non-partisan and representative public service, able to provide services in both official languages, in which appointments are based on the values of fairness, access, transparency and representativeness.

The Public Service Modernization Act (PSMA) (2003) and the PSEA (2005) increased the accountability for human resources (HR) management in federal organizations.⁴ While deputy ministers are formally accountable for HR management within their organizations, there are many other organizations involved, including the Office of the Chief Human Resources Officer at the Treasury Board Secretariat and the Canada School of Public Service.⁵

The PSC works with others in government who share responsibility for the public service, notably: the Clerk of the Privy Council, Treasury Board and federal organizations (with delegated responsibility for recruitment and appointment). In response to the PSEA, the Program's business model was re-aligned to reflect the PSC's new role as an optional, common service provider and to assist federal organizations to meet their recruitment responsibilities as well as support the government's Public Service Renewal Action Plan.⁶

In 2006, the Prime Minister's Advisory Committee on the Public Service was established to advise the Prime Minister and the Clerk on public service renewal. In addition to the Advisory Committee, the Deputy Minister Committee on Public Service Renewal was established to advise the Clerk on modernizing the business of government to ensure the 21st century public service reflects excellence and leadership at all levels.⁷

In his Fifteenth Annual Report to the Prime Minister, tabled in March 2008, the Clerk established 14 specific Public Service Renewal commitments.⁸ With respect to recruitment, the Clerk set a goal of hiring 3 000 indeterminate post-secondary graduates for 2007-2008 and then increased that target to 4,000 appointments for both 2008-2009 and 2009-2010. The number of graduates recruited through the PSR Program⁹ has fluctuated over the years in relation to the needs of the public service (Exhibit 5.1).

Exhibit 5.1: Post-Secondary Recruitment Appointments (1973 to 2009)

Exhibit 5.1 long description

Source: PSC, Post-Secondary Recruitment to the Public Service (2002) and PSC Annual Reports

The Program was redesigned in 2006-2007, introduced the testing of applicants in the general inventory and offered tested career choices as part of the Program. Since the redesign, PSR appointments have increased from 450 in 2006-2007 to 1,698 in 2008-2009.

Interviewed PSR and PSC managers view the Program as supporting federal recruitment needs for entry level officers by providing a centralized mechanism that aggregates supply and offers potentially huge economies of scale through uniform recruitment and testing of post-secondary graduates. They note that the PSR addresses all current priorities and needs: PSEA requirements, the NAoS¹⁰ guidelines, the Clerk's priorities, the need for transparency and fairness in recruitment, and flexibility to meet organizational requirements.

However, interview respondents underscore that the relevance of the PSR Program to government recruitment needs and priorities is limited by its under-utilization. Limited use of the Program, particularly in the regions, has dampened its potential contribution to recruitment needs. This was also emphasized in the findings of the 2000 Office of the Auditor General's audit that recommended increased use of the Program.¹¹ Two regional managers interviewed also note that few tested career choices are outside the National Capital Region and that there is little engagement of the regions in terms of exploring their recruitment needs. In the words of one respondent "the Program is Ottawa-centric". Some interview respondents feel the under-utilization of the PSR Program by hiring managers contributes to applicants' frustration and dissatisfaction as they sit in the General Inventory for extended periods of time.

Several PSC and PSR managers interviewed believe that the extent to which the PSR supports recruitment needs has increased over time with the introduction of testing, the tested career choices and the Clerk's priorities (which have increased use). Only one PSC manager interviewed does not believe that the PSR Program is supporting government recruitment needs. This respondent felt that in an environment of delegated responsibility, a centralized approach to entry-level recruitment is irrelevant in that each department can undertake their own recruitment.

The perceived relevance of the PSR Program also varies between interview respondents who have used it in the past and those who have not. Past users agree that the PSR Program is highly relevant to their organization and supports government recruitment priorities, even in an environment of delegated responsibility. One client notes that the PSR provides a "one-stop recruitment point for students and departments" which "removes the burden from departments". This respondent further notes that without the PSR Program, it would be costly, time-consuming and would involve extensive duplication to meet the Clerk's priorities.

On the other hand, non-users of the PSR Program are not convinced that it is particularly relevant to their needs. These managers believe that they have specific requirements that cannot be met by a centralized recruitment function. Needs they do not perceive the PSR Program to meet include: specialized skill sets or degrees, a need to fill positions quickly given operational requirements, a preference for individuals with experience and to hire students who have already held work terms in their organization.

Key Findings: Under the PSEA, the PSC is mandated to support staffing in the public service. While the PSMA and the PSEA have shifted accountability for HR management to hiring organizations, the PSC remains mandated to safeguard the integrity of the public service staffing system and the political neutrality of the public service. Public service renewal has been underscored as a priority on numerous occasions, not least through the establishment of the Prime Minister's Advisory Committee on the Public Service, and the recruitment targets set by the Clerk.

The PSR Program is designed to provide hiring departments with a coordinated, consistent, efficient and national approach for recruiting post-secondary graduates. However, the relevance of the PSR Program is perceived to be limited by its under-utilization. A perceived lack of relevance of the PSR Program among some hiring managers who have never used it may be contributing to this lack of use.

5.2 Design and Delivery

Under the issue of design and delivery, the evaluation explored various facets of the current PSR Program design, including the approach to screening, the timing of the annual campaign and the focus on university recruitment. Additionally, the evaluation examined the effectiveness of current communication and the user-friendliness of the Program. Evaluation questions are addressed through results from all surveys and findings from interviews with key personnel.

Question 2: Is the current PSR Program design meeting the needs of hiring managers throughout the federal government?

General Interests and Preferences of Hiring Managers

The needs or preferences of hiring managers were explored in the on-line survey of hiring managers. They were asked to rate the importance of several aspects when staffing an entry-level officer position. Among the aspects examined (in making hiring decisions), personal suitability factors and the abilities of candidates are assigned highest importance (94%), followed by reference checks (84%). Organizational test results (55%) and PSR standardized test results (46%) are viewed as less important criteria.

In interviews with key personnel, hiring managers (n=8) emphasize the importance of having as much flexibility and control over the recruitment process as possible and being able to conduct their own assessment and screening of candidates.

Need for Pre-Assessed College Graduates

The PSR annual campaign has traditionally focussed on universities although the PSC has also recruited from colleges by using tested career choices to meet the needs of clients. The survey of hiring managers explored their need for college graduates. The majority of the hiring managers (57%) indicated that they do not need pre-assessed college graduates; however, a sizeable minority (43%) did. This need is most keenly felt by those who have been responsible for staffing in the federal government for over 10 years (49%) as well as by managers who plan to staff three or more positions this year.

Those who indicated a need for pre-assessed college graduates were asked, unprompted, for what types of jobs they would require college graduates. Chief among these jobs are analyst, officer level, Information and Technology (IT) and administrative positions.

Timing of the Post-Secondary Recruitment Campaign

Hiring managers were also surveyed to rate their satisfaction with the timing of the annual PSR campaign. Results suggest general satisfaction with this issue: 44% of the hiring managers reported being satisfied with the timing of the campaign, 32% were neutral (neither satisfied nor dissatisfied) and only 13% were dissatisfied.

Only a few interview respondents expressed dissatisfaction with the timing of the campaign. Three of eleven university representatives interviewed suggested that the timing is not in line with career fairs and on-campus recruitment, which tend to occur in January and February (and hiring decisions are made in April or May at the end of the academic year). Two HR advisors interviewed suggest that the timing does not meet the needs of organizations that are generally seeking to hire in the fall. It is important to note that the preference of hiring managers does not coincide with the timing suggested by university representatives. However, some PSC managers considered the timing as somewhat immaterial, given that the inventory is maintained year-round and that referrals can be made at any time. Also, these PSC managers note that the campaign already receives more applicants than needed, suggesting that the timing is working appropriately.

In terms of timing the PSR campaign, some interview respondents (hiring managers and HR advisors) describe recruitment to the public service to be "a lengthy process", with one stating "there is a long wait for PSR candidates". Several university representatives interviewed also describe the hiring process through the PSR Program as "lengthy", with a "long time lapse from submission to selection".

Potential Modifications to the PSR Program

The survey of hiring managers included several forward-looking questions aimed at obtaining suggestions for improving the PSR Program or gauging support for potential Program modifications.

More than half (57%) of the hiring managers surveyed say that they would be more likely to use the PSR Program if it included language testing. Almost half (48%) also say they would be more likely to use the Program if it included either changes to increase the availability of mid-career professionals or year-round intake and testing. Over four in ten (43%) feel that they would be more likely to use the Program if it included a general inventory of college graduates.

Alignment of the Campaign with Hiring Managers Needs

The concern mentioned most frequently regarding the current design mentioned relates to the current alignment between applicants and recruitment needs. PSC managers interviewed feel that the Program is not sufficiently restrictive in determining who can be tested and included in the inventory and that there should be closer alignment between the needs of hiring organizations and both the number and type of applicants tested. The hiring managers and HR advisors interviewed also believe that the PSR Program cannot meet all their specific needs in terms of the skills and areas of study required (including some specific areas of need for college graduates) and their need for some candidates with experience. As well, two HR advisors identified a specific need for employment equity candidates and a concern that these are not sufficiently targeted or identified by the PSR Program.

Key Findings: Hiring managers place high importance on personal suitability, abilities and reference checks (that are not easily assessed through centralized testing and screening) and place less emphasis on test results. Furthermore, over 50% of the hiring managers place high importance on experience when hiring at the entry level, suggesting some disconnect between how an "entry-level" recruit is defined by the PSC and by hiring managers.

Most survey and interview respondents appear satisfied with the timing of the campaign, although a minority express an interest in either a year-round campaign or a change in the timing. Over half the hiring managers surveyed suggested that they would be more likely to use the Program if language testing were included and over 40% stated an interest in a mid-career stream, year-round intake and a general inventory of college graduates. Over 40% of hiring managers surveyed also identify a need for pre-assessed college graduates in a range of jobs or positions including analysts, officers, IT and finance and administration.

Hiring managers interviewed echo the findings of the survey, expressing some concern with the alignment between their entry level recruitment needs and PSR applicants. These managers indicated an interest in candidates with experience and specific skills or fields of study, suggesting that the Program may benefit from incorporating these components into the screening process.

Question 3: Is the current approach resulting in effective and appropriate screening of referrals for departments and regions across the federal government?

The PSR Program currently uses three standardized tests in its screening process: the Situational Judgement Test (SJT), the Graduate Recruitment Test (GRT) and the Written Communication Proficiency Test (WCPT). All applicants to the general inventory complete these tests. Testing undertaken for career choices varies. Organizations can post untested career choices or can tailor testing to their needs. The perceived accuracy of tests in assessing skills was explored both with employees recently hired through the PSR Program and with hiring managers.

Hiring Managers Perceptions of the Accuracy of Test Assessments

Hiring managers surveyed (n=519) express somewhat mixed views on the accuracy of the PSR standardized tests. The highest positive ratings are given to the WCPT (48%). The GRT and the SJT received lower positive ratings (38% each).

However, results reveal high proportions of "don't know/not applicable" responses to these questions. Examining the results based on valid responses only (i.e., with the "don't know/not applicable" category removed and n=118), reveals that a majority of hiring managers who have used the PSR Program feel that these tests provide an accurate assessment of applicants' abilities and skills (Exhibit 5.2).

Exhibit 5.2: Accuracy of Assessment Provided Through PSR Tests (Valid Responses)

[IF RECRUITED/HIRED THROUGH THE PSR PROGRAM]

"In your view, how accurate an assessment of applicants' abilities and skills are provided through each of the PSR standardized tests?

Exhibit 5.2 long description

Perceived Accuracy of Tests mong Post-Secondary Recruits

Employees hired through the PSR Program were also asked how accurate an assessment of their abilities and skills each test provided. As with hiring managers, the WCPT is rated highest in terms of perceived accuracy in assessing skills among PSR recruits (59%). The SJT and the GRT are seen as less accurate (43% and 39%, respectively).

Use of Standardized Tests in Screening – Key Informant Interviews

The current test-based approach to screening was explored in interviews with key personnel. University representatives interviewed express satisfaction with the administration of testing but are unable to comment on the effectiveness of this approach to screening.

PSC management and hiring managers interviewed express mixed views on the tests as an accurate indicator of performance. Three interview respondents who have used the PSR in the past (HR advisors and hiring managers) believe that these tests are a good indicator of performance and found that their own assessment of candidates was closely correlated to scores. Other respondents are more interested in personal suitability factors (including communication and teamwork skills) that are not assessed through standardized tests. One manager expressed concern that they sometimes re-interview some candidates several years in a row, given that they have the highest test scores (although they do not find them suitable). Another hiring manager interviewed stated that he was not allowed to see test scores or to conduct further screening on candidates once provided PSR referrals. Further investigation found this proved to be only a perception, given that managers can, in fact, examine testing scores.

Hiring managers interviewed who have never used the PSR Program express dissatisfaction or disagreement with the use of standardized testing for screening. These respondents clearly have the impression that they do not have control over screening as the hiring manager, stating that "I want to have control over testing and screening" or "I don't know those three tests and I'm not convinced that I want them". Others state "I want to be able to do my own screening, such as obtaining samples of their writing", or "PSR may be creating a pool of qualified candidates, but qualified for what?"

Some regional managers express concern that many students are unable to attend testing and are removed from the process (one region noted that only 60% of interested applicants were able to write the tests this year). As well, testing is a gruelling process for students and an administrative burden for regions to coordinate. Some PSC managers note that testing plays an important role in volume management. Many managers feel that there is a need for a more cost-effective and applicant-friendly approach to screening. The possibility of on-line testing is considered very attractive; however PSR Program managers note that developing an on-line version of the SJT will require 18 months. Others also suggest restricting the number and type of applicants as an approach to improving efficiency.

One HR advisor who had used both the tested career choices and general inventory found that the general inventory was of little value without subsequent language testing. He had received numerous referrals from the general inventory to staff bilingual positions in the NCR but found that he had to start his own external recruitment process, given that none of the candidates were sufficiently bilingual. The fact that language testing is only provided on a cost recovery basis was perceived as a barrier in this instance.

Other Types of Tests to Evaluate Entry-Level Officer Candidates

Hiring managers responding to the on-line survey were also asked, unprompted, if there are any other tests that could be used to evaluate entry-level officer candidates. Among the few suggestions that were provided, psychological testing and position-specific testing were mentioned most often.

Despite few top-of-mind (i.e., unprompted) suggestions regarding alternate tests to evaluate entry-level officer candidates, when prompted about potential improvements to the PSR Program, a majority of the hiring managers surveyed feel that language testing (reading and writing) would make them more likely to use the Program (see previous section).

Key Findings: Most hiring managers surveyed who have used the PSR Program consider that the three standardized tests provide a fairly accurate assessment of candidates' abilities, although many lack sufficient knowledge of the tests to be able to rate them. Recruits are somewhat less positive in their assessment of the three tests. The WCPT is rated most highly as an assessment tool by both hiring managers and recruits surveyed. Not all hiring managers interviewed feel that these tests provided a good assessment of abilities, due to their interest in other factors such as personal suitability.

Most PSC managers believe that there is a need to improve the cost-effectiveness of the current approach to screening, given the large volumes tested annually and the relatively small number hired.

Hiring managers interviewed who have never used the PSR Program (and are thus unfamiliar with the tests) disagree with the use of standardized testing for screening due to a perception that this may reduce the control they have over the screening process

Question 4: Is the Program and process user-friendly for hiring managers? Is it perceived as user- friendly by applicants?

Interviewed hiring managers and HR advisors did not raise any concerns regarding the user-friendliness of the PSR Program and process. Those who have not used the PSR Program exhibit some confusion and misunderstanding of the Program and how it functions, but these perceptions are largely based on a lack of knowledge and do not reflect concerns regarding user-friendliness. The primary concern regarding the process is linked to the length of time required to complete the recruitment process.

Surveys of applicants indicate high satisfaction among those recently hired through the PSR Program across a number of the process-oriented dimensions but notably lower satisfaction levels for overall fairness and feedback on the status of applications. Satisfaction is highest in terms of the ease of finding the Web site (77%), ease of navigating the site (68%), clarity of information and instructions (66%) and ease of completing the application forms (61%). While satisfaction with the time required to complete an application is positive on balance (54%), dissatisfaction is also relatively high (30%). Satisfaction with the overall fairness of the PSR Program is also relatively low: only about half say they are satisfied with this aspect (51%), while nearly a quarter (23%) are dissatisfied. At the bottom of the list is satisfaction with the feedback provided on the status of applications: only a third (33%) say they are satisfied and half (50%) are dissatisfied with the feedback received.

Satisfaction with Hiring Process and Reasons for Dissatisfaction

New employees hired through the PSR Program and other methods were asked a series of identical questions about their satisfaction with the hiring process. Overall, satisfaction levels are largely the same between those hired through the PSR and through other methods. A majority is satisfied with the hiring process (53 % of those hired through the PSR and 55% of those through other methods). However, a sizeable minority is dissatisfied (34% and 31%, respectively).

Those who expressed dissatisfaction with the PSR hiring process were asked, unprompted, to explain the reasons for their dissatisfaction. A poor or deficient testing method (34%) and the lengthiness of the process (31%) are cited as the top reasons. A perceived lack of communication is also cited by a smaller number of these respondents (17%).

Among recent hires recruited through other methods, dissatisfaction is largely focussed on the length of time associated with the hiring process (55%). A number of these respondents also felt that the process was unfair (16%) or too difficult (11%).

Two-thirds of the university campus representatives interviewed express concerns with the user-friendliness of the PSR Program and application process. On several campuses, respondents find that they have to set up appointments or run workshops to explain the application forms and process to students. Concerns include the following:

• Lack of clarity in information communicated: Several respondents note that the information on the site is not always clear. For example, students experience confusion over the difference between the tested career choices and general inventory, what happens once the on-line registration is completed and also may not be aware that there is a registration deadline for testing.
• Complexity and length of the application process: Interview respondents consider the application process to be complex and lengthy, from the extensive on-line application forms to the in-person testing. This process was described as "daunting" or as a "disincentive to some". One respondent states "it is often the first experience students have with the federal government as an employer and it is not a good one - tends to add to stereotypes of bureaucracy". Another notes "the process is much more complex than most of the opportunities available to students".
• Lack of feedback: Another concern expressed by a few respondents is the lack of feedback received on applications. One respondent notes that "students can wait for months and then receive an e-mail for a job posting for a department they have never heard of before".

The remaining one-third of university representatives interviewed note that they have received little feedback on the PSR Program from students and assume that it is sufficiently user-friendly, given the absence of complaints. Some of these interview respondents also note that the low ratio of hires of PSR applicants limits the satisfaction (and potentially perceived fairness) among applicants.

Key Findings: Overall, applicants surveyed are satisfied with the process-oriented dimensions of the PSR Program (e.g., ease of finding and navigating the Web site, clarity of instructions, ease of completing the form), but are less satisfied with the feedback received on applications, the fairness of the process and the time required to complete an application. In interviews, some university representatives describe the process as "complex" and note that low success rates may contribute to perceived unfairness.

Based on survey results, satisfaction with the hiring process is not markedly different between recent recruits hired through the PSR Program and those hired through other methods or processes. Those dissatisfied with the PSR process most often identify the testing, the length of the process and communication as sources of dissatisfaction.

Question 5: How effective are current communication and outreach activities undertaken by the PSR Program?

The PSR Program involves communication, outreach and marketing to internal (hiring managers and HR advisors) and external (universities and students) stakeholders. In surveys and interviews, the Program's communication and outreach activities were examined from both perspectives.

Communications and Outreach to Potential Applicants

Results from the survey of "potential" recruits (i.e., university graduates under 35 years of age who have not previously worked for the federal government) reveal that awareness of the PSR Program is relatively low (75%).

Satisfaction with information available on jobs.gc.ca was explored with recent hires through the PSR Program and other non-PSR methods. Satisfaction with this information is high among both respondent groups: over three-quarters of recent hires through the PSR Program and other methods say they are satisfied with the information available on this Web site (78% each).

Communication issues were explored in interviews. There seems to be significant variation in the level of communication provided to campuses across Canada. Roughly half of university representatives indicated that promotion and communication of the PSR Program by the PSC is limited. They noted that they receive little or no promotional material related to the campaign (e.g., brochures or posters) and that there is no PSR "presence on campus". One respondent describes it as "overall, an inferior branding and marketing strategy". Others note that information on the PSR Program, the deadlines involved and the overall application process are not well communicated to students. These respondents express interest in seeing greater communication efforts to promote the PSR campaign on campus. Two university representatives noted that campaign promotion is inconsistent: some years they receive material to support them and others they do not. Three other university representatives interviewed express satisfaction with the promotional material and support they receive from the PSC. It should also be noted that the 2000 audit of the PSR Program identified the need for more systematic and aggressive use of university recruitment campaigns.¹²

When asked to comment on the level of awareness among students of the PSR campaign, several university representatives note that students are very interested in the Government of Canada as an employer. Despite this, several note that the lack of presence on campus or promotion of the federal government as an employer somewhat dampens this interest. These respondents express an interest in seeing greater on-campus presence to promote the campaign and to "field questions from students". Some note that there is a perception of government as "a good employer but that it's hard to get in". One notes that there is "little confidence that they will be hired".

While some university representatives interviewed express dissatisfaction with the promotion of the PSR campaign, several PSR managers interviewed note that, given the high volume of applicants, there is little need to undertake additional promotion. Applications have risen from 45 352 in Fall 2007 to 95 368 in Fall 2009.

Communications and Outreach to Potential Users

Survey results reveal fairly low levels of knowledge about the PSR Program among hiring managers. Almost half of the hiring managers surveyed (45%) indicate little knowledge of the Program and only one in four (27%) feel they are knowledgeable.

This lack of knowledge was also evident in the follow-up interviews conducted with managers who had responded to the survey (n=8). Some of these managers are uncertain as to whether they had ever used the PSR and asked for it to be described to them. Two others have used the PSR Program in the past but are uncertain as to whether they had used the general inventory or tested career choices. One interview respondent stated that they did not know or understand the PSR Program and saw "no appeal in trying to figure out a new process". Several managers (both users and non-users) interviewed indicate that they typically turn to their HR advisor for advice on selecting the appropriate recruitment process.

Survey results further reveal that hiring managers express fairly negative views on the effectiveness of the PSR Program's current communication and outreach efforts. Only one in seven judge these efforts to be highly effective and almost three times as many (37%) say these efforts are not effective (although it should be noted that those who have recruited through the PSR Program in the past are much more likely than those who have not to rate PSR communications as effective). These findings are echoed by respondents to interviews with key personnel. Most PSC managers and hiring managers interviewed feel that communication is of limited effectiveness.

While efforts are made to make HR advisors and hiring managers aware of the PSR Program (e.g., a letter to deputy heads announcing the availability of the general inventory), they are not believed to be sufficient. Several note a need for a more comprehensive marketing approach to explain the purpose of the PSR Program, its potential benefits, how it can be used, options available and tools to use. Some note that even though potential users may be aware of the existence of the Program, they do not understand its full scope or how it can be useful to them. Another respondent suggests that there is confusion between different programs such as PSR, pools and the Federal Student Work Experience Program (FSWEP) and a need to explain the features of each.

Key Findings: Awareness of the PSR Program is low, both among hiring managers and among potential recruits. In addition to a lack of general awareness, some hiring managers appear to lack knowledge of the PSR Program or confuse it with other programs.

Hiring managers do not rate the effectiveness of current PSR communication and outreach very highly and the assessment of the effectiveness of communication is markedly lower among hiring managers who have never used the Program.

In terms of communication to applicants, satisfaction with information available on jobs.gc.ca is high among recent recruits. Communication to potential applicants via university campuses appears to be inconsistent. Many university representatives note a lack of presence on campus to promote the PSR campaign, however PSC managers note that the volume of applicants is so high that further promotion is unnecessary.

5.3 Outcomes/Success

In examining Program success, the evaluation explores both the use of the Program and its outcomes. In terms of use, the evaluation explores past and future use of the Program by hiring managers as well as factors that may impede or encourage its use. In terms of outcomes, the evaluation then examines satisfaction with referrals provided by the Program as well as with candidates referred. The evaluation also examines the extent to which the Program provides post-secondary graduates with access to job opportunities in the public service, contributing to a competent and representative public service and supporting government-wide renewal priorities and needs.

Question 6: To what extent do hiring departments and agencies throughout the federal government use the PSR Program? Are there any factors that either impede or facilitate its use? Is the Program being integrated into organizational HR planning?

Past Use of the Post-Secondary Recruitment Program

Exhibit 5.3 provides PSR Program data on applications from the last three campaigns. In 2008-2009, a total of 1,697 PSR candidates were appointed by hiring organizations from a total of 35,967 referrals derived from 24,757 unique applicants. Over the three-year period, a total of 85,478 unique applicants were tested and just under 2,681 post-secondary graduates were hired through the PSR Program from 2007 to 2008. Significant increases were noted between 2006-2007 and 2008-2009 in the annual number of applicants and appointments into the public service.¹³

Based on Exhibit 5.4 below, in 2008-2009, 72% of federal appointments of PSR candidates were in the NCR, compared to 79% in 2006-2007. While in absolute numbers,¹⁴ appointments of PSR candidates in the regions increased during the three year period from 94 appointments in 2006-2007 to 474 appointments in 2008-2009, the relative level of representation remained stable for all provinces and territories (with the exception of Ontario, where the percentage of annual PSR appointments increased from 7% in 2006-2007 to almost 12% in 2008-2009). The three Territories were the least represented regions with a total of eight PSR candidates appointed to the public service over the three-year period reviewed.

Figures may not add up to 100% due to rounding.

Source: PSC Annual Reports to Parliament (2006-2007 to 2008-2009)

Use of the Post-Secondary Recruitment Program

Hiring managers surveyed were asked a number of questions regarding the use of the PSR Program. Results reveal that the majority of the hiring managers surveyed (65%) say they have not recruited / hired through the Program and only about one in three (35%) say they have. Among hiring managers who have hired through the PSR Program, most say they have used the General Inventory (67%), while close to half say they have used a career choice where testing was not administered (48%) and 39% say they have used a tested career choice.

Survey results also reveal that relatively few hiring managers (27%) are planning to use the PSR Program to fill their entry-level officer recruitment needs in the next year. The likelihood of using the Program is significantly higher among those who have used it in the past (52% compared to 14% who have not used it before), as well as among managers working in the NCR (33%).

Post-Secondary Recruitment Proposed by Human Resources Advisors

Hiring managers interviewed identified their HR advisor as an important source of advice and information on recruitment, suggesting that decisions regarding recruitment are made in partnership with their HR advisor. Given the low level of past or planned use of the PSR Program among surveyed hiring managers, it is perhaps not surprising that on-line survey results indicate that it is not uniformly proposed by organizational HR advisors. Only 53% of hiring managers indicate that the PSR Program is proposed by their HR advisor as a possible recruitment tool for entry-level officer positions at the time when the selection process is initiated. Even fewer hiring managers (42%) say that their HR advisor proposes the PSR general inventory option.

Perceptions Regarding what would Impede or Encourage use of the PSR Program

Hiring managers participating in the on-line survey were asked an open-ended question on what would make them more likely to use the PSR general inventory and / or tested career choices in the future. Over half of the hiring managers surveyed (53%) did not provide a response to this question, suggesting a lack of familiarity with the Program. Among those who provided a response, the most frequently mentioned suggestion pertained to increasing awareness of the PSR Program (17%). A number of these respondents also cited improvement in the ease of use of the Program, improved tools and processes for matching candidates to criteria / positions and better access to candidates (8% each).

Use of the PSR Program and factors that may impede its use were also discussed in interviews with key personnel. A number of factors are cited as impeding potential use of the PSR Program, including:

• Limited awareness: Low awareness and understanding of the PSR Program was identified as a factor in its under-utilization. However, many caution that awareness is not the only factor. In the words of one respondent, "increased awareness alone will not increase uptake".
• Potential mismatch between screening and needs: Interview respondents indicate that they often require candidates with experience, either at the entry or mid-career level and that this need is not met by the PSR Program. As well, some hiring managers suggest that they want "proven" candidates and thus prefer to use bridging.
• Lack of alignment between needs and applicants: Again, interview respondents note that they or hiring managers generally may need specific skill sets or degrees that they believe they cannot find through the PSR inventory. Examples provided include chaplaincy candidates or candidates with mediation skills, engineers, IT candidates from college programs and employment equity candidates. Several interview respondents identify a need to create a stronger link between HR planning and the PSR Program for it to be more useful.
• Perceived lack of control and flexibility: Interview respondents, including hiring managers, identify a need for "control" over the recruitment process, noting that they know and understand what types of candidates they are seeking and that they are not interested in candidates screened through a centralized program. Many hiring managers believe that skills they require (e.g., leadership, interpersonal, communications) are not being adequately assessed through standardized testing. In some instances there is a perception that they will not have control over the process and there is little understanding that they can subsequently screen candidates according to their own requirements.
• Cost of national recruitment: Several hiring managers and regional PSC managers identify the national basis of recruitment as a factor potentially impeding use. Several hiring managers interviewed state that "I can't afford to fly students in for interviews, or to pay to move candidates". Several regional managers also note that "hiring managers don't want to entertain students from other regions". One further notes "they may say they're willing to move but they're only interested in a central location".

Key Findings: Use of the PSR has increased significantly in the past three years in terms of number of hires made. Based on statistics available, use of the PSR Program is predominantly in the NCR (roughly three-quarters of hires, and in 2008-2009 two-thirds of the appointments were made through tested career choices (with the balance through the general inventory).

Despite increased use, only one-third of surveyed hiring managers say they have used the PSR Program to recruit, although, in contrast to the statistics described above, these respondents say they are more likely to use the general inventory than the tested career choices (suggesting that hiring managers may confuse tested career choices with Departmental Programs and may be unaware of their actual usage in the PSR Program). Furthermore, survey results reveal that the Program is often not proposed as a possible recruitment tool by HR advisors when initiating the selection process for entry-level positions. Planned future use of the Program is low: only 27% of surveyed hiring managers say they plan to use it in the next year.

Factors potentially impeding use identified in interviews include lack of awareness, unmet needs (e.g., candidates with experience and perceived lack of control and flexibility over the screening and hiring process when using PSR). Consistently, perceptions of factors that might encourage use (as identified by hiring managers) include greater awareness, increased flexibility or ease of use, improving the match of candidates to criteria and access to what managers need.

Taken together, these findings suggest that the PSR Program is not well integrated into HR planning by departments and agencies.

Question 7: Is the PSR Program resulting in timely, appropriate and useful referrals to hiring federal organizations throughout the government?

These evaluation questions are examined by exploring the number of referrals made, satisfaction with referrals and satisfaction with candidates.

Statistics on Referrals

As noted, in 2008-2009, 1,697 PSR candidates were hired from a total of 18,610 referrals, for a ratio of 1 hire to 11 referrals (Exhibit 2.5). The annual ratios of hires to referrals through the PSR Program were constant for 2007-2008 and 2008-2009. Hires to applicants declined steadily over the three-year period reviewed, from 1 hire to 37 applicants in 2006-2007 to 1 hire to 16 applicants in 2008-2009.

Source: PSC Annual Reports to Parliament (2006-2007 to 2008-2009)

Satisfaction with Referrals and Candidates

Hiring managers surveyed who indicated they had recruited / hired through the PSR Program were asked about their satisfaction with the overall quality and number of candidates and with the referrals received. The highest levels of satisfaction are found with respect to the quality of candidates (79 % are satisfied), followed by the number of candidates (63% satisfied). Just over half (54%) indicate satisfaction with general inventory referrals and just over four in ten are satisfied with the tested career choice referrals (44%). However, results also reveal a fairly high proportion of non-response across these questions, particularly in terms of satisfaction with the general inventory and tested career choice referrals. Examining the results, based on valid responses only, reveals fairly high levels of satisfaction across the areas examined (at least two-thirds or more express satisfaction when looking at the valid responses to these questions).

All hiring managers were asked to rate the extent to which they agree that candidates referred by the PSR Program are a good match for job criteria (please note that there was a high level of non-response to these questions). Again, looking at the valid responses to these questions, just under half of the hiring managers surveyed agree that candidates referred from the PSR general inventory are generally a good match (46 %) and that candidates referred by the PSR for tested career choices are a good match (47%) for job criteria.

Satisfaction with referrals and candidates was also explored in interviews with HR advisors and hiring managers who had used the PSR Program in the past (n=6). In general, interview respondents are satisfied with the speed and quality of referrals they received. One manager in particular notes that the candidates have performed very well to date, while another describes the quality of the candidates as "very good". However, one (using the tested career choice) notes that they receive a larger quantity of referrals than they need or want and that they are looking for approaches to further reduce the number interviewed in proportion to the number of positions available. Another expressed dissatisfaction that they see some of the same candidates year after year and have to fly them in for an interview even though they performed poorly on past interviews.

Key Findings: In 2008-2009, the ratio of hires to referrals was 1 to 11. Hiring managers surveyed who have used the PSR Program appear generally satisfied with the quality and number of candidates they receive as well as with referrals. However, less than half of the hiring managers surveyed agree that candidates referred are a good match for job criteria (although these managers may be basing their views on perceptions, as not all have used the Program).

Question 8: Is the PSR Program providing post-secondary graduates across Canada with access to job opportunities in the public service? Is it attracting the right type of candidates to the federal government?

To address this evaluation question, the evaluation examined the extent to which potential recruits are applying to the public service for employment and the method they used. In addition, perceptions of the public service and broader career aspirations and values were explored among potential recruits, those hired through the PSR Program and those hired through other methods.

Application for Jobs in the Public Service

The survey of potential recruits explored the extent to which they have applied for work in the public service and reasons they have not. Potential recruits were first asked if they had ever applied for a job in the federal public service. Results reveal that most of these respondents (67%) say they have not applied for a job in the public service. Those who did apply for a job were asked, unprompted, how they applied for a position. The PSR Program is mentioned most often (33%), followed by the jobs.gc.ca Web site (27%). Therefore, while many potential recruits have not applied for a job in the federal government, among those who have, the PSR Program is used most often as the application method.

Belief that the PSR Program provides Job Opportunities in the Public Service

Survey results indicate that the majority of recent hires through the PSR believe the Program provides university graduates with access to job opportunities in the public service. Over half (55%) feel strongly that the PSR Program provides university graduates with access and a further 3 in 10 (28%) feel this is the case to some extent. Only 16% feel the PSR Program provides little access to job opportunities in the federal government.

Perceptions of the Public Service

Potential recruits to the federal government, recent hires through the PSR Program and recent hires through other methods were all asked to indicate their level of agreement with a number of statements about the public service. Overall, the public service is viewed favourably on a number of key employment indicators such as good work-life balance, a wide variety of careers and opportunities for advancement. There are, however, high levels of agreement that it is both difficult and time-consuming to obtain employment in the public service. Results also reveal that perceptions of the public service are more positive among those working in the federal government than among potential recruits. Furthermore, among recent recruits, those hired through the PSR Program hold more positive views than do those recruited through other methods. These results are displayed in detail in Appendix C.

Importance of Job Aspects

Potential recruits to the federal government, recent hires through the PSR Program and recent hires through other methods were also asked to rate the importance of a variety of job aspects. Results suggest that those hired through the PSR Program tend to be more career-oriented. They are more likely than either of the other two groups to see opportunities for advancement and prestige as important job attributes and they are less likely to see working in a location where they want to live as important (suggesting they are more willing to move for their job). Results also reveal that employees hired through non-PSR methods are more likely than those hired through the Program to place greater importance on quality of life, income and security related-issues (e.g., a good work-life balance, job security, compensation and a pension). These results are displayed in detail in Appendix D.

Values and Risk-Taking

Potential recruits, recent hires through the PSR and recent hires through other methods were asked what values should shape the federal government's direction. Overall, the three groups share a very similar value structure, suggesting that the public service is largely representative of the broader Canadian population. There are, however, some noteworthy differences: recent recruits to the public service are more likely than potential recruits to rate integrity as an important value for the government. New hires are also more likely than potential recruits to feel that tolerance and diversity, Canadian identity and equality among regions are important values to guide the government. These results are displayed in detail in Appendix E.

Potential recruits, federal employees hired through non-PSR methods and those hired through the PSR Program were also asked to assess their sense of confidence and their risk-taking behaviour. Results are largely similar among the three groups with some notable differences. Interestingly, potential recruits are more inclined than either of the employee groups to choose a salary increase over a bonus if they had a one in three chance of receiving the bonus (suggesting higher levels of risk aversion than the employee groups). However, they are also more likely to prefer intense disappointment over intense regret (suggesting higher levels of risk-taking behaviour on this question). It is also noteworthy that recent hires through the PSR Program are significantly less likely than the other groups to agree with the idea that if forced to choose, they would take safety over achievement, reinforcing other results that suggest PSR recruits have more of a career focus than the other groups surveyed. These results are displayed in detail in Appendix F.

Belief that New Entry Level Officers Reflect Values of the Public Service

Survey results reveal that a majority of hiring managers believe that new entry-level officers reflect the values and goals of the public service to a great (41%) or moderate (30%) extent, whereas only one in five hiring managers (18%) think that new entry-level officers do not reflect the values and goals of the public service. The belief that entry-level officers hired by the government reflect the values and goals of the federal public service is more likely to be held by those who have recruited through the PSR Program.

Key Findings: While the survey of potential recruits found that relatively few have applied for a job in the public service, those who had had done so through the PSR Program. Most recent hires through the PSR Program agree that the Program provides access to jobs in the public service.

Potential recruits view the government favourably on a number of key employment indicators (although those already hired hold more positive views) and recent recruits hired through the PSR Program hold more positive views of federal employment than do those recruited through other methods. Results also suggest that those hired through the PSR Program tend to be more career-oriented than the general population and recent hires through other methods and most hiring managers agree that new entry-level officers reflect the values of the public service.

Taken together, these results suggest that the PSR Program is providing access to job opportunities in the public service and is attracting high calibre applicants.

Question 9: Is the PSR Program supporting government-wide renewal priorities and hiring manager needs? To what extent is the Program contributing to a competent and representative public service, through recruitment based on fairness, access, transparency and representativeness?

For 2008-2009,¹⁵ departments and agencies reported that more than 4,200 post-secondary graduates had been appointed or offered appointments, of which just under 1,697 (40%) had been referred through the PSR Program.

Hiring managers who were familiar with the PSR Program were asked attitudinal questions regarding the extent to which it is attracting the candidates needed (as with many other questions, a high proportion of the hiring managers surveyed did not provide a response to many of these questions). Looking at the valid responses, over half of hiring managers agree that the PSR Program helps them meet public service renewal needs and targets (55%). Moreover, three-quarters agree that candidates hired through PSR have generally performed well in positions for which they were recruited (76%) and 6 in 10 agree that candidates hired through the PSR Program have generally been promoted within the public service (59%). It is important to note that hiring managers that have recruited through the PSR Program in the past are much more likely to hold positive perceptions of the Program than those who have not.

The perceived success of the PSR campaign in providing graduates with access to job opportunities in the public service and success in attracting a high calibre of talent was also explored in interviews with university representatives. University representatives noted that students are generally very interested in public service jobs and therefore attracted by the PSR campaign. Many suggest that the intensive process (e.g., testing, followed by interviews) is designed to ensure a high calibre of talent. One notes that "candidates have to be serious to give up a day for testing". However, several believe that the calibre of talent attracted may be limited because public service job offers are typically made later than in the private sector (suggesting that the best and brightest have already been "snapped up"), especially given the lack of "buzz" or personalized attention on campus around the PSR campaign.

Interviewed university representatives and PSC managers agree that the Program is designed to contribute to a competitive and representative public service through recruitment based on fairness, access and transparency. They note that the PSR Program is transparent and fair, in that anyone can apply and be tested. Furthermore, applicants are invited to self-identify as employment equity candidates, making it possible to recruit on that basis. However, some respondents are uncertain of the extent to which candidates self-identify. Some university representatives suggest that more involvement of representatives on campus during the campaign (including representatives of equity groups) might encourage applications by employment equity candidates. As well, one noted that the on-line application process discourages Aboriginal candidates because it is not a user-friendly interface.

Based on available PSR program data, the linguistic profile of PSR applicants remained stable over the three years reviewed, with 65% indicating English and 35% citing French as their maternal language.¹⁶ Similarly, applicant demographics were constant over the period reviewed (Exhibit 5.6). Applicant demographics were higher than workforce availability (WFA) for individuals that self-identified as belonging to a visible minority but were somewhat lower than WFA for Aboriginals, persons with a disability and women.

Source: PSC Annual Reports to Parliament (2006-2007 to 2008-2009)

The Area of Selection Policy was amended in December 2008 to expand the NAoS in all external advertised appointment processes to all occupational categories in all regions across the public service. The PSR Program is delivered through the national e-recruitment system, the Public Service Resourcing System, which ensures NAoS compliance.¹⁷ In the 2009-2010 Public Service Renewal Action Plan,¹⁸ in collaboration with the Office Chief Human Resources Officer and deputy ministers, the PSC committed to make its job-seeker Web site fully operational to enable all interested Canadians to apply to any job open to external candidates, regardless of the region in which they reside.

Key Findings: In 2008-2009, 1,697 of the 4,000 post-secondary jobs targeted for graduate recruitment by the Clerk were filled through the PSR Program. A majority of surveyed hiring managers perceive the PSR Program as addressing their renewal needs and targets and managers who have used the Program are particularly likely to hold positive views. Findings also indicate that candidates hired through the PSR Program are perceived to perform well in the positions for which they were recruited.

Interview respondents agree that the Program is designed to contribute to a competitive and representative public service, although some note that additional effort could be made to encourage employment equity candidates. The PSR Program also respects the requirements of NAoS. The demographics of PSR applicants have remained stable over the past three years. Visible minority applicants to the PSR Program were higher than WFA but lower for Aboriginal candidates and persons with disabilities.

Question 10: Is the PSR providing an effective centralized mechanism to recruit new entry-level officers? Is it eliminating duplication in staffing entry-level officer positions? Has the Program had any unintended impacts or outcomes, either positive or negative?

Hiring managers surveyed were asked to rate the extent to which they use a variety of methods for recruiting recent graduates. Results reveal that among the methods examined, the most prevalent are student bridging (45%), followed by general recruitment (40%). The PSR Program is used to a great extent by only about one in three (36%) of surveyed hiring managers, 24% say they advertised a specific position, 12% say they use the general inventory and 33% indicated they use departmental programs to a great extent when recruiting university graduates.

Alternative approaches used for entry-level recruitment were also explored via interviews. As with survey results, the primary alternatives identified by interviewed hiring managers and HR advisors were bridging, direct external recruitment (jobs.gc.ca), regional pools and on-campus recruitment with their organization or with TBS. Several managers note that their organization occasionally participates in job fairs on campuses that have educational programs of interest to them. None identified alternative departmental programs. However, as mentioned, some did express confusion with the PSR Program and were uncertain as to whether they had used it in the past.

University representatives interviewed also note that students see CO-OP appointments and bridging "as another way in" to the public service. They note that a lack of awareness of the fall campaign, lack of feedback on applications, confusion over different government programs (e.g., PSR, FSWEP, CO-OP), lack of PSR presence on campus and that many organizations hire independently may all be factors that encourage students to seek alternatives.

Statistics on Recruitment of Post-Secondary Graduates

The number of appointments to the public service increased from 17 700 in 2006-2007 to 23 750 in 2008-2009,¹⁹ for a 25% increase in federal appointments during the three year period reviewed, compared to an overall increase in PSR appointments of 73% (Exhibit 5.7).

Figures may not add up to 100% due to rounding.

Source: PSC Annual Reports to Parliament (2006-2007 to 2008-2009)

From 2007-2008 to 2008-2009, 713 more appointments were made through the PSR Program or a 72% annual increase, while FSWEP new hires stayed relatively steady at 22% of the total, despite the increase in absolute numbers of 236 new hires for a 26% year over year increase. The data suggests that while the PSR Program is making inroads into the potential market, in 2008-2009, there was still room to increase market share in this previously unexploited area by 2,337 post-secondary new hires.

Figures may not add up to 100% due to rounding.

Perceptions of PSR vs. non-PSR Programs

Hiring managers who used a non-PSR method when recruiting university graduates were asked if they felt non-PSR approaches better met their hiring needs than the PSR Program. Results suggest that a slight majority of these hiring managers (56%) feel that non-PSR approaches meet their hiring needs better. This view is significantly more prevalent among hiring managers who have not recruited through the PSR Program.

Similarly, when asked to rate their agreement with the statement "I would prefer to use my own recruitment process than use the PSR Program", 44% of the hiring managers surveyed agree with this statement and only 25% disagree.

In interviews conducted with hiring managers who had never used the PSR Program (n=4), there was some discussion of why the PSR Program is perceived as less preferable to alternatives. These interview respondents noted that the PSR would have to be faster (compared to alternatives such as bridging), more flexible responding to their needs and requirements and that they would require access to language testing for bilingual positions. Several noted that they are not interested in "flying in" candidates from across the country and lack the financial resources to do so.

Unintended Impacts

No significant unintended impacts were identified through the evaluation.

Key Findings: Consistent with previous findings that indicated a low level of use of the PSR Program, findings suggest that the PSR Program is not used consistently as the primary central mechanism to recruit new entry-level officers. Hiring managers more often report using student bridging and general recruitment than the PSR Program when recruiting recent graduates. While data on the numbers of entry-level officers recruited through other methods is not available, there appears to be greater focus on student hiring through FSWEP and CO-OP (which are sources of bridging candidates) than on the PSR Program.

5.4 Cost-effectiveness and Alternatives

Under the issue of cost-effectiveness, the evaluation examined the costs involved in the PSR Program and perceptions of cost-effectiveness. However, it must be noted that an absence of data on costs associated with alternative recruitment methods limits the analysis performed. Finally, information on selected international public service recruitment programs is examined for comparison purposes.

Question 11: Is the PSR Program providing the PSC value for money? How could the efficiency or return on investment of the PSR Program be improved, if at all?

Post-Secondary Recruitment Program Costs

Exhibit 5.9 summarizes average per unit costs for PSR activities. In 2008-2009, the average cost per unique applicant was $51 and the average cost per test administered was $39. These costs were consistent with 2007-2008 per unit costs. The average cost per referral declined from $81 in 2007-2008 to $74 in 2008-2009 for an overall reduction of $7 over the two years (9%). The reduction in campaign costs (excluding testing fees) as well as the increased number of PSR hires resulted in significant reductions in the average cost per PSR hire from $1,475 in 2006-2007 to $1,136 in 2007-2008 and to $801 in 2008-2009 for an overall reduction of 84% in the average cost per PSR hire.

Source: PSC Annual Reports to Parliament; PSR Program Data (2006-2007 to 2008-2009)

Data on costs associated with alternative recruitment approaches for entry-level hires is not available, limiting the potential cost-effectiveness analysis.

Cost Recovery

Services provided through the PSR Program (including general campaign costs, all PSC assessment, special accommodation, referrals and general inventory) are not provided on a cost recovery basis.²⁰ The cost effectiveness of the PSR Program could potentially be improved through the introduction of service fees to hiring organizations. While 65% of direct program costs for both 2007-2008 and 2008-2009 represented testing fees charged to the Program by PSC assessment services, under the current business model there is no cost for departments and agencies to participate in annual PSR campaigns. In their 2009-2010 Report on Plans and Priorities, the PSC committed to providing modernized staffing and assessment services on an expanded cost recovery basis.²¹

Perceived Cost Effectiveness

Survey results reveal that over half of the hiring managers surveyed (54%) agree with the idea that the PSR Program offers a cost-effective approach to hiring entry-level positions, compared to departmental / agency recruitment programs.

The issue of cost effectiveness and the value for money provided by the PSR Program was explored in interviews with key personnel. PSR clients interviewed (HR advisors and hiring managers) describe the PSR Program as "providing excellent value for money" and note that they are "glad that it is free". Several stated they are uncertain they would continue to use the Program if it was offered on a cost recovery.

PSC managers interviewed note that the Program is not as cost-effective as it could be, given the number of applicants tested compared to the number hired. Again, these respondents underscore the need to bring tested applicants in line with hiring needs (in terms of numbers and types of applicants).

Key Findings: The limited data available on PSR Program costs and of alternative recruitment approaches limits the cost-effectiveness analysis that can be performed. PSR clients interviewed feel that the PSR Program provides them with excellent value for money. However, the fact that services are largely provided for free is an important factor and interview respondents are not certain they would continue to use the Program if cost recovery were introduced. PSR managers interviewed note that the Program is not as cost-effective as it could be, given the number of applicants tested compared to the number hired.

Question 12: Are there other, potentially more effective approaches to facilitating post-secondary recruitment for the public service? How does the PSR Program compare to other post-secondary recruitment programs? Within Canada? Internationally?

Potential Alternatives

The introduction of on-line, non-proctored testing is anticipated in the future.²² To date, the PSC has developed two non-proctored tests, a cognitive test and a second language self-assessment test, which are administered on-line through the Public Service Resourcing System (PSRS). Although not developed specifically for the PSR Program, individuals applying for specific positions will be asked to complete the non-proctored cognitive test as part of the application process. The PSC intends to provide additional on-line staffing tools to support deputy ministers in meeting Public Service Renewal objectives and to improve staffing efficiency and effectiveness.²³ The transition to non-proctored testing should facilitate the participation of post-secondary graduates from all regions as well as reduce costs for rental accommodations, travel and non-professional services.

University representatives interviewed believe that the PSR Program is comparable to the recruitment processes used by large private sector employers (e.g. Procter & Gamble) or by other large public sector employers (e.g., the Royal Canadian Mounted Police). Several others suggest, however, that other employers tend to have a greater "presence" on campus, meeting students and promoting themselves as employers. Again, one interview respondent notes that the private sector can make job offers more quickly, which may also make it more difficult to compete for "the best and brightest".

Comparison with Human Resources and Skills Development Canada Recruitment

The PSC conducted a comparison of post-secondary recruitment between Human Resources and Skills Development Canada (HRSDC) and the PSR Program, based on post-secondary recruitment conducted by HRSDC for a PM-2 in 2007 and two ES-2/3s in 2007 and 2008.²⁴ While the data compared was not identical in that different assessment tools were used, it provides a comparison with the PSR Program. HRSDC posted these positions on jobs.gc.ca and used a TBS (using the PSRs) and manual screening approaches to select candidates. Exhibit 5.10 provides comparison data on recruitment by HRSDC with the PSR campaign (see Exhibit 5.3 for greater detail), based on applications received and number of candidates who would have been available, and on the campaign costs.

This comparison demonstrates that while HRSDC undertook a campaign to obtain applicants to their specific positions, an ample supply of potential candidates would have been available from the PSR general inventory. It is more difficult to compare the costs incurred based on the fact that the HRSDC costs include the costs to screen and interview applicants, whereas additional organizational-specific screening and interviews are not included in the PSR campaign costs. However, the comparison indicates that costs and duplication could have been avoided by HRSDC by accessing candidates within the general inventory as opposed to undertaking a separate campaign. In their analysis,²⁵ the PSC furthermore concludes that:

• Using the PSR general inventory would reduce the time to staff for hiring managers; and
• The cost efficiency of the PSR Program augments as the number of organizations using it increases.

International Comparisons

As a comparison, the evaluation also examined several international recruitment programs in Australia, the United Kingdom and the United States. The graduate recruitment programs in these three countries are contrasted. The recruitment programs examined place greater emphasis on training and development and are more similar in focus to the Recruitment of Policy Leaders Program in that they screen candidates and provide them subsequent development experience.

Australia's Public Service Commission has the Graduate Development Programme which includes an extensive learning and development component combined with on-the-job training. The learning and development program spans 11 months and comprises an introduction / formalized training, at least three work rotations and support or mentoring. Upon successful completion, graduates are placed in intermediate- level positions. In addition, the Australia Public Service offers cadetships that target university recruits. These are administered directly by hiring government agencies and generally involve work placements during full-time study.

The United Kingdom's Civil Service Fast Stream Program operates two core components with specific professions (e.g., science and engineering, diplomatic service, economists, etc.). The selection and assessment process spans approximately seven months and consists of on-line tests, followed by on-site assessments and selection. Prior to the annual campaign, participating departments and agencies determine the number of vacancies to be filled. Successful candidates are assigned to a specific department that is responsible for their training and development, are offered several work assignments in different areas and are actively encouraged to pursue secondments.

Recruitment of recent graduates into the United States public service is undertaken through two internship programs, the Presidential Management Fellows Program and the Federal Career Intern Program. In both programs, successful candidates are required to complete a two year training and development program before being eligible for permanent placement. Appendix G describes this information in more detail.

Key Findings: One improvement to facilitating post-secondary recruitment for the public service is the introduction of on-line or non-proctored tested (currently under development). While recruitment programs in Australia, the United Kingdom and the United States provide an interesting contrast, these programs are quite different in that they include a training and development component and only offer successful candidates employment following several work placements.

6. Recommendations

Overall, evaluation findings suggest that the Post-Secondary Recruitment (PSR) Program is generally meeting its objective of being a national recruitment program for the federal government. It fits into the Public Service Commission's (PSC) broader mandate of supporting staffing in the public service and its use among hiring managers is increasing. In addition, the PSR Program is viewed quite favourably among both recent hires recruited through the Program and among hiring managers (particularly those who have used the Program in the past).

However, the evaluation does point to areas of possible improvement across three key areas: communication, outreach and marketing, testing and assessment and alignment with hiring needs.

Communication, Outreach and Marketing

1. Increase the focus on marketing the PSR Program to both hiring managers and human resources (HR) advisors within the National Capital Region and in the regions:

• Planned future use of the Program remains low and increased efforts to promote the Program and communicate its benefits are needed, particularly in the regions.
• There is a need to ensure that hiring managers are aware of the various recruitment tools available for student and / or graduate hiring. Distinctions between the recruitment programs (e.g., PSR, FSWEP, Co-op/Internship program, etc.) need to be clarified. There is a lack of understanding of what the PSR Program is, the value it can offer hiring managers, when and how it can benefit them and how to use it.
• There is a need to emphasize the flexibility and control managers can have using the PSR Program and to address and overcome factors impeding its use. Given the interest in flexibility and a tailored focus, there is also a need to increase promotion of the availability of tested career choices.
• There is a need to promote that subsequent screening, language testing and other assessments are available on a cost recovery basis when accessing the general inventory.

2. Increase clarity in communication tools for example, with respect to the steps in the process, for ease of use applicant and ease of understanding by university / college institutions (Web site, outreach, presence on campus, etc.). Increase the focus on targeted marketing to potential applicants in shortage groups with a view to attracting the right applicants. A more consistent approach to the marketing of the Program across regions is also needed to ensure national access. This could include:

• Seeking applicants in the fields in demand (whether at universities or colleges); and/or
• More consistent and targeted marketing of the Program to universities and more applications from target employment equity groups.

Testing and assessment

3. Explore the complement of tests to better control costs and increase efficiencies. This could involve:

• Ensuring that tests used for screening applicants best fit the job requirements (i.e., not all tests are relevant to all jobs). Hiring managers and applicants felt that the Written Communication Proficiency Test was the most useful indicator of candidate performance.
• Controlling costs by exploring methods to reduce the number of candidates tested.

4. Explore options for on-line or unproctored Internet testing to increase efficiency and facilitate the process for applicants.

5. To better meet hiring managers' preferences, consider Program alterations to include additional elements such as experience and language testing, on a cost recovery basis.

Greater alignment with hiring needs

6. Explore methods to better align the number and type of applicants with the needs of hiring departments / agencies and hiring managers. These could include:

• Increasing the linkages of the PSR Program to HR planning and working in partnership with hiring federal organizations to ensure that the annual campaign is focused in areas of their need.
• Exploring redesign options for the general inventory to more specifically meet organizational and regional needs.
• Identifying experience as a field in the application process to enable managers to screen on this requirement.

7. Evaluation Limitations

To the extent possible within the available budget and timeframe, the evaluation methodology incorporated multiple methods and data from different primary and secondary sources in order to ensure that the findings were valid and captured key points of view on the Post-Secondary Recruitment (PSR) Program. It is important to note, however, that there are some limitations to the evaluation evidence. These limitations include the following:

• Survey of PSR recruits limited to hires: The survey conducted of recent recruits to the public service through the PSR Program was limited to hires and did not include tested applicants available for hire through the PSR Program. As such, any findings from this survey cannot be generalized to all applicants.
• Confusion between tested career choices within PSR and organizational programs: From results of the survey of hiring managers and interviews with key personnel, it is clear that the PSR Program is not well understood by hiring managers. In particular, some hiring managers surveyed appear to have mistaken tested career choices for a departmental / agency recruitment program. As a result, use of the Program is likely under-reported in the survey.
• Limited cost-effectiveness data: A lack of data on comparative programs and approaches limits the cost-effectiveness analysis that can be performed in this evaluation.
• Lack of data on PSR applicants: While statistics are available on the number of applicants to the PSR Program and on the number of hires made through the Program annually, there is little profile data available on applicants, such as the regional breakdown of applicants and data on the general inventory versus tested career choices. This limits the conclusions that can be made regarding the reach of the Program.

8. Management Response and Action Plan

Appendix A: Logic Model / Intended Outcome

Logic Model / Intended Outcomes

A logic model illustrates how the activities of a policy, program or initiative are expected to lead to the achievement of the final outcomes. The logic model serves to identify the short, medium and long-term outcomes of a policy, program or initiative. In that sense it serves as a road map for the policy, program or initiative.

A logic model is a visual representation that links activities, outputs and outcomes relevant to a program and for which a performance framework is being developed. Since measurement of program performance is tied to program objectives and because the logic model articulates how a program intends to achieve its objectives, the logic model can also provide the foundation for developing a set of key performance indicators.

The exhibit on the following page presents a logic model for the Post-Secondary Recruitment (PSR) Program.

Post-Secondary Recruitment Program Logic Model

See also: Logic Model / Intended Outcomes - graphic version

Logic Model / Intended Outcomes long description

Appendix B: Evaluation Matrix

Appendix C: Perceptions of the Federal Public Service

"To what extent do you agree or disagree with each of the following statements about the federal public service?" [Open]

Appendix C long description

Appendix D: Importance of Various Job Aspects

"How important are each of the following when deciding what type of career you might want to pursue?"

Appendix D long description

Appendix E: Values

Preferred Values and Goals in Shaping Federal Government

"If you were to direct the federal government as to which goals or values should be most important in shaping its direction, how important would you say each of the following goals or values should be?"

Appendix E long description

Appendix F: Self-Rated Confidence and Perceptions of Risk-Taking

"To what extent do you agree or disagree with each of the following statements?"

Appendix F long description

Appendix G: International Comparisons

Description of International Recruitment Programs
Australia, United Kingdom and United States

Australian Public Service Commission Graduate Development Programme

The Australian Public Service (APS) Commission Graduate Development Programme²⁶ includes an extensive learning and development program combined with on-the-job training. Administered and managed by the APS Commission, the Graduate Development Programme provides successful candidates with the opportunity to complete a Diploma in Government and several electives from the Commission's public program learning and development calendar. In addition to this training, selected graduates participate in the Commission's internal learning and development program.

Interested candidates can apply for a graduate position with the APS Commission by downloading an application kit from the Commission's Web site. There are three parts to the application kit: a personal details form, an application letter and academic transcripts. During the first two weeks of August graduates can submit their completed hard copy application for admission to the Graduate Development Programme. Applicants selected for interview are contacted in August with interviews held in the capital, Canberra, in September. Travel and accommodations are provided for out-of-rstate applicants selected for interview. Successful applicants are contacted in October and made an offer of employment. All graduate employees in the development program are relocated to Canberra to join government agencies. The Commission provides relocation assistance and temporary accommodations for successful candidates.

The learning and development program runs for 11 months and is composed of three key components: a comprehensive induction and formalized training program; at least three diverse and challenging work rotations and support from a range of staff, including previous graduates. On successful completion of the Graduate Development Programme, recruits are placed in intermediate-level positions within the APS. Although recruitment does not appear to be linked to organizational vacancies or functional requirements, each year over 500 successful applicants are appointed into the Graduate Development Programme. The Social Network of Graduates (SNoG) is a social network that connects recruits and alumni of the program within the public sector. SNoG organizes orientation and networking events that provide graduate recruits with an opportunity to meet with individuals from across the Australian public service.

In addition to the Graduate Development Programme, the APS offers three non-graduate entry-level programs that are, in most cases, administered directly by hiring government agencies:

• School leaver programs – these programs are for people who have recently completed high school (Grade 12). They may include work placements within various areas of the agency and, in some cases, may also include tertiary study if it relates to the position.
• Traineeships – are suitable for school leavers and employed individuals. They usually involve work experience with a government agency and the opportunity to gain a formal qualification.
• Cadetships – these programs recruit university students to start working while they finish their degrees. They generally involve full-time study with work placement during school breaks. In most cases, agencies provide some form of financial assistance towards the participants' educational costs.

In addition to these entry-level programs, recruitment initiatives are undertaken on an ad hoc basis. These initiatives may be run by a single agency, a group of agencies or across the whole APS. Campaigns provide a link to recruitment initiatives such as graduate programs, large bulk recruitment rounds or specialist skill campaigns to recruit particular skills that are needed within the APS.

United Kingdom – Civil Service Fast Stream

The Civil Service Fast Stream is an accelerated training and development program for entry-level graduates.²⁷ Administered by the Cabinet Office, the Fast Stream is a single source program that guides and directs graduates through the civil service recruitment and appointment process. The civil service recruits to the Fast Stream on the basis of fair and open competition and selection on merit. Fast Streamers are selected for their potential to become leaders of the future and many are expected to reach the senior civil service. The Civil Service Fast Stream program comprises two core components: the Graduate Fast Stream and Fast Stream Schemes (professional entry routes). The Graduate Fast Stream component is composed of five career options: central departments (Home Civil Service), science and engineering (Ministry of Defense and Department for Business, Innovation and Skills), diplomatic service, European (EU civil servants) and Houses of Parliament. The Fast Stream Schemes (professional) component includes the Economist scheme, Statistician scheme, Government Communications Headquarters scheme (Graduate Management Trainees) and Technology in Business scheme. In addition to the Civil Service Fast Stream program, undergraduates are encouraged to gain experience or accept term placements with the civil service before graduating through three subsidiary programs: sandwich courses, vacation visits and work experience.

The annual Graduate Fast Stream selection and assessment process takes approximately seven months to complete and consists of on-line tests, an on-site e-Tray exercise, on-site assessments, selection and pre-appointment checks. All Graduate Fast Stream candidates who achieve the pass mark during the on-site assessment component have reached the required standard and are guaranteed a place in central organizations. The other Fast Stream options apply some form of final selection procedure (selection board or merit ranking) before recommending a candidate for appointment. Candidates who are interested in joining the civil service but who are unsuccessful in either the on-line tests, the e-Tray exercise or the on-site assessment are redirected back to the government's on-line job database. There is also an annual competition (the In-Service competition) for serving civil servants who are interested in entering the Fast Stream program. The table below provides an overview of the Graduate Fast Stream application and selection process.

Prior to the annual campaign, participating departments and agencies determine the number of vacancies to be filled for each of the Graduate Fast Stream options and the Other Fast Stream schemes, with an average of 400-500 graduates annually. Successful candidates of the Graduate Fast Stream are assigned to a specific department or agency that is responsible for organizing individual training and development. While they are in the program, Graduate Fast Stream recruits are provided with 12-18 month assignments and gain experience in at least two of three main work areas of the civil service: policy, operational delivery and corporate services. There are also secondment opportunities available with most departments and agencies and they are actively encouraged. Successful candidates of the Fast Stream Schemes component are assigned to trainee programs or professional development programs within specific departments and agencies, such as: Government Economic Service, Government Social Research, Government Statistical Service, Government Operational Research Services and Government Procurement Service. In addition to the Civil Service Fast Stream program, some government organizations offer their own graduate recruitment schemes such as: Government Finance Service, Government Legal Service and Government Actuaries Department.

In addition to information posted on the Civil Service Fast Stream Web site and in their youth-oriented promotional material, a number of career events are undertaken in support of outreach, including:

• Career fairs – occur in spring and summer (April to July) and in the autumn (October to December) and provide visitors with a unique opportunity to speak directly to civil service employees about working in the civil service, applying to the Fast Stream and how the selection process works.
• Presentations – the Fast Stream marketing team organizes a program of presentations with black and Asian societies across the country and offers the opportunity for alumni to return to universities and share their experience with the Fast Stream program with current undergraduates.
• Dunchester's Millions (role-play policy game) – a half day policy game that enables participants to learn about how government operates and, in particular, the role that civil servants play. The game also provides participants with an opportunity to practice a range of skills including effectiveness in meetings, persuasion and negotiation, analysis, decision-making and innovation.

United States – Presidential Management Fellows Program and Federal Career Intern Program

Recruitment of recent graduates into the American federal public service is undertaken through two key internship programs: the Presidential Management Fellows Program²⁸ and the Federal Career Intern Program²⁹. For both programs, successful candidates are required to complete a two-year training and development program before being eligible for permanent placement within a federal agency. In addition to the Presidential Management Fellows Program and the Federal Career Intern Program, there are a number of employment and educational opportunities available to students high school through doctorate level and career professionals, including apprenticeships, cooperative education, fellowships, grants, internships and scholarships. Information and access to specific programs and participating agency Web sites is provided through a one-stop portal "STUDENTJOBS.gov" under the "e-Scholar" umbrella term.

Presidential Management Fellows Program

Administered by the Office of Personnel Management (OPM), the Presidential Management Fellows (PMF) Program assists agencies to meet their management succession planning needs by providing the next generation of federal managers and executives. The PMF Program is highly selective, with only 1 in 10 applicants being chosen for a two-year Fellowship. The PMF Program accepts only individuals who demonstrate a breadth and quality of accomplishment, a capacity for leadership and a commitment to excellence in the leadership and management of public policies and programs. With its focus on developing future leaders, the PMF Program has several advantages for agencies over traditional graduate recruitment methods, notably:

• Top graduates – OPM only accepts applicants nominated to the PMF Program by the Dean, Chairperson or Academic Program Director of their schools.
• Pre-qualified candidates with superior skills – OPM rigorously assesses each nominee to ensure finalists have superior writing, interpersonal, analytical and leadership skills.
• Year-round hiring – OPM hosts a searchable on-line résumé bank and job-posting system that assists federal agencies to hire Fellows.
• Developing new leaders – OPM sponsors orientation programs and training fora for all Fellows.

Presidential Management Fellows are available for hire year-round and hiring managers search an on-line résumé bank of PMF finalists, conduct interviews and can make employment offers in a matter of days, saving considerable time and expense. When agencies hire a PMF Fellow, they commit to developing the leadership potential of high achieving individuals within a wide range of academic areas, including business, health, human resources, law, public administration, science and technology. Agencies that hire a PMF Fellow agree to:

• Hire the Fellow on a two-year appointment;
• Establish a performance plan and conduct an annual evaluation;
• Create an Individual Development Plan with the Fellow;
• Provide 80 hours of formal classroom training for each year of the fellowship (160 hours in total);
• Place the Fellow in a developmental assignment of 4 to 6 months that provides responsibilities in line with the Fellow's target position;
• Evaluate, through the Executive Resources Board (ERB), whether the Fellow successfully completed the program requirements; and
• Offer the Fellow a permanent full-time position at the end of the Fellowship, if ERB certified.

OPM works with hiring agencies to provide intensive training, mentoring and hands-on experience to selected Fellows. After PMF Fellows have completed the two-year program, they are assigned permanent positions. Every participating agency has an Agency PMF Coordinator who serves as a liaison and program facilitator between the agency, Fellows and their supervisors and the PMF Program Office. OPM delivers the PMF Program on a cost recovery basis and participating federal agencies' PMF fee includes:

• Recruitment, nomination, assessment and qualification for an appropriate entry-level position;
• Agency participation in the annual PMF Job Fair for finalists;
• PMF Orientation and Training Program;
• Other training opportunities, such as fellowship fora;
• Graduation ceremony for Fellows;
• On-line access to post positions and / or search résumés; and
• Training, support and assistance for agencies.

Federal Career Intern Program

The Federal Career Intern Program is designed to help federal agencies recruit and attract exceptional individuals into a variety of occupations. In general, selected applicants are appointed to a two-year internship. Upon successful completion of the internships, the interns may be eligible for permanent placement within an agency. Initial appointments can be made at any grade level for which the agency has a formal training program in place. Interns can be appointed year-round and there is no limitation on promotion potential. The Career Intern Program requires a two-year formal training and development program, but does not require participants to go on rotational assignments.

Although subject to OPM qualifications, there is no nominating process for the Career Intern Program and applicants are not required to have a university degree. Individuals interested in Career Intern opportunities must contact specific agencies directly. Candidates must be screened using OPM-approved assessments to evaluate applicants. Agencies that have been trained and certified by OPM may administer the approved assessments directly by the agency's certified Test Security Officer.

Endnotes

1. Sixteenth Annual Report to the Prime Minister on the Public Service of Canada, Clerk, Privy Council Office, March 2009 [Return]

2. Fall 2008 Post-Secondary Recruitment Campaign Approach - Strategic Discussion Paper, September 2008 [Return]

3. PSC 2009-2010 Estimates, Report on Plans and Priorities, February 2009 [Return]

4. Annual Report to Parliament 2006-2007, People Management in Transition, Canada Public Service Agency, 2008 [Return]

5. Report to the Prime Minister, Prime Minister's Advisory Committee on the Public Service, March 2007 [Return]

6. PSC Annual Report to Parliament 2006-2007 [Return]

7. The DM Advisory Committee advises the PSC on issues concerning the delivery of its staffing and assessment services, including services provided through the PSR Program. [Return]

8. Fifteenth Annual report to the Prime Minister on the Public Service of Canada, Annex 2: Progress on Public Service Renewal Since the 2007 Report, March 2008 [Return]

9. Post-Secondary Recruitment to the Public Service, PSC, June 2002; PSC Annual Reports to Parliament (2006-2007 to 2008-2009) [Return]

10. Implementation of the NAoS means that all externally advertised jobs are open to all Canadians, regardless of their area of residence [Return]

11. House of Commons Committees - PACC (37-1) - Post Secondary Recruitment Program of the Federal Public Service of the December 2000 Report of the Audit [Return]

12. Report of the Auditor General of Canada – December 2000 Chap. 21 pp.1-30. [Return]

13. PSC Annual Reports to Parliament (2006-2007 to 2008-2009); see Appendix A Table 3 [Return]

14. PSC Annual Reports to Parliament (2006-2007 to 2008-2009); see Appendix A, Table 1b [Return]

15. Sixteenth Annual Report to the Prime Minister on the Public Service of Canada, Clerk, Privy Council Office, March 2009 [Return]

16. PSC Annual Reports to Parliament (2006-2007 to 2008-2009) [Return]

17. PAC Annual Report to Parliament 2008-2009 [Return]

18. 2009-2010 Public Service Renewal Action Plan, Privy Council Office, 2009 [Return]

19. PSC Annual Reports to Parliament (2006-2007 to 2008-2009); see Appendix A, Table 2a [Return]

20. Post-Secondary Recruitment (PSR) and FSWEP Update, Presentation to PSC DM Advisory Committee, Nov., 2009 [Return]

21. PSC 2009-2010 Estimates, Report on Plans and Priorities, February 2009 [Return]

22. Post-Secondary Recruitment and FSWEP Update, Presentation to PSC DM Advisory Committee, November 2009 [Return]

23. PSC 2009-2010 Estimates, Report on Plans and Priorities, February 2009 [Return]

24. Post-Secondary Recruitment (PSR). A comparison between HRSDC and PSC. PSC, December 9, 2009. [Return]

25. Ibid. [Return]

26. Australian Public Service Commission Graduate Development Programme, 2009 (www.apsc.gov.au/apscrecruitent/graduates.html) [Return]

27. Fast Stream (www.civilservice.gov.uk/jobs/FastStream/index.aspx); Civil Service Fast Stream Reports (www.cabinetoffice.gov.uk/reports/faststream.aspx) [Return]

28. Presidential Management Fellows Program, U.S. Office of Personnel Management (www.pmf.opm.gov) [Return]

29. Federal Career Intern Program, U.S. Office of Personnel Management (www.opm.gov/careerintern) [Return]