Evaluation of the Public Service Employee Survey
Executive summary
The Public Service Employee Survey (PSES) has been conducted every three years since 1999. The Treasury Board of Canada Secretariat (the Secretariat) managed four of these iterations, including the two latest cycles. The most recent iteration, with data collection in 2014 and the release of results in 2015, had 106 questions and was completed by 182,165 employees.
This evaluation of the PSES examines the survey’s relevance and performance, with input from the following:
- interviews with departmental and agency heads, Secretariat executives, and other stakeholders
- a survey of PSES primary users
- a review of PSES documents
- a trend analysis of PSES results
- an examination of public service employee surveys in other jurisdictions
Conclusions
The evidence shows the continuing need for the PSES as a reliable and comprehensive tool to periodically collect opinions from public service employees about their jobs and their workplace. The PSES addresses government priorities regarding its support for an effective public service, and administration of the PSES fits within the Secretariat’s mandate.
The PSES offers valuable insight into employees’ opinions about their engagement on the job, the federal workforce and workplace, and leadership in the public service. It also helps identify concerns regarding these areas. However, more could be done to provide a deep understanding of the root causes of these concerns, in particular those related to leadership. Federal executives across the Government of Canada regularly use PSES results in their departmental and agency plans for improving people management practices. To a lesser extent, PSES results contribute to the Secretariat’s development of the government’s policies for people management.
While the PSES has contributed to initiatives that aim to enhance public service employees’ engagement and performance, there is limited evidence of government-wide responses to address issues identified in the PSES. In parallel, while government-wide PSES results show positive change over time for some areas, there is also negative and little to no change over time in several others.
The areas of performance management, a respectful workplace and an ethical workplace have improved over time, while results for job satisfaction, satisfaction with organization, career development, and organizational performance have weakened over time. Little to no change has occurred in results on senior management, official languages, staffing and labour relations.
The PSES is effectively designed and administered, and there have been systematic efforts to continuously improve it.
Notwithstanding this, there are four significant concerns regarding the PSES program:
- the frequency of the PSES’s iterations
- the time lag between fieldwork and the release of results
- limitations in the analysis of PSES data
- adequacy of government-wide support to address identified issues
Although determining the PSES’s cost-effectiveness was beyond the scope of this evaluation, interviewees indicated that the PSES provides adequate value for money. Statistics Canada administers the PSES, although there may be alternatives from suppliers in the private sector that are more economical.
Recommendations
- The Secretariat should develop a strategy to proactively address government-wide issues highlighted in the PSES results. Such a strategy should include the Secretariat:
- playing a stronger leadership role in government-wide initiatives for change while respecting individual deputy head accountabilities
- bringing greater consequences to departments and agencies regarding their PSES performance, depending on their level of improvement
- taking specific actions to address those issues that could benefit from government-wide intervention.
- The Secretariat should develop a plan to enhance the PSES program that addresses the following:
- providing timelier insights to departments and agencies on employee engagement, the workplace, the workforce and leadership issues
- facilitating ongoing consultation with organizations and continual improvement of survey tools and products
- enabling a wider range of analysis, including collecting additional targeted data
- ensuring that the PSES represents the state of the art in survey methodology and remains pertinent to key user and stakeholder groups
Introduction
The Public Service Employee Survey (PSES) has been conducted every three years since 1999. There have been six iterations of the PSES to date.
As part of the Treasury Board of Canada Secretariat’s (the Secretariat’s) five-year evaluation plan, it evaluated the PSES over the summer and fall of 2016. This evaluation presents findings on the relevance and performance of the PSES program and presents considerations for its future.
Although this evaluation examines the PSES since its inception, it focuses particularly on how the 2014 iteration performed in its program design and short-term outcomes.Footnote 1
For a summary of the PSES’s background, including its governance, roles, process and resources, refer to Appendix A.
Purpose and logic model of the PSES
The expected immediate outcome of the PSES is that insight is gained into employees’ opinions on employee engagement, leadership, the workforce and the workplace, across the public service and at the level of departments and agencies.
The immediate outcome is expected to lead to the following intermediate outcomes:
- current and accurate information is available to inform people management policies across the public service
- plans are developed that address issues and opportunities identified by the PSES
Intermediate outcomes are expected to lead to the long-term outcome of improvements that lead to a high-performing and engaged public service that delivers on government priorities now and in the future.
The long-term outcome relates to the Secretariat’s strategic outcome of “good governance and sound stewardship to enable efficient and effective service to Canadians.” The causal relationships between these elements are presented in the PSES logic model in Appendix B.
Evaluation of the PSES
This report is the first independent evaluation of the PSES. Its purpose is to assess the PSES program’s performance and continued relevance, and to explore possible alternative approaches to its design and delivery.
Evaluation scope and questions
The evaluation examined the PSES from its inception in 1999 to its most recent iteration in 2014. In its examination of program design, activities, outputs and immediate outcomes, the evaluation focused on the 2014 PSES. Although the evaluation questions included relevance and performance considerations taken from Treasury Board guidance,Footnote 2 emphasis was on the program’s performance and its design and delivery, with a view to suggesting improvements for future iterations. The complete list of evaluation questions appears in Appendix C.
Evaluation methodology
The evaluation is based on multiple sources of evidence:
- a review of 41 documents, including PSES foundational documents, a sample of departmental and agency Management Accountability Framework reports, central agency reports, and PSES activity reports
- a trend analysis of PSES results
- an examination of public service employee surveys in Canada’s provinces and territories, and in the US, the UK and Australia
- interviews with 10 deputy heads, 10 Secretariat executives and 9 stakeholders
- a survey of 166 primary users of the PSES (representing a 72.8% response rate)
The methodology is described in more detail in Appendix D.
Relevance
Conclusion: The evidence supports the continued need for the PSES as a reliable and comprehensive tool to periodically collect opinions from public service employees about their jobs and their workplace. The PSES addresses government priorities regarding its support of an effective public service, and administration of the PSES fits within the Secretariat’s mandate.
Continued need for the PSES program
To what extent is the PSES aligned with current and emerging needs with respect to assessing the state of employee engagement, the workforce, the workplace and leadership?
Interviewees were virtually unanimous in stating that the PSES fulfills a critical need to regularly assess employees’ engagement and their views on key aspects of their jobs and their workplace. Interviewees spoke of assessing “the public service culture,” the “mood” of the workforce and the level of “positivity,” along with desired public service attributes such as inclusivity. Senior departmental and agency officials noted the need for survey findings to allow them to compare their organization with others, and to measure their organization’s progress over time. Union officials noted the need to support bargaining efforts, public awareness campaigns and dialogue with the employer.
Emerging needs include gaining a better understanding of employee morale, mental health and wellness, harassment, flexible work arrangements, LGTBQFootnote 3 issues, and generational issues. Interviewees also spoke of the need to have a better understanding of the workforce’s capacity to cope with downsizing and workload. Some of these issues are already being addressed in the PSES; for example, seven new questions will be added to the 2017 PSES on mental health.
There were 90% of primary user survey respondents who reported undertaking additional survey activities in their organizations based on the PSES. Such activities included internal follow-up surveys and focus groups, and other activities geared to better understanding PSES results (84% of respondents) and obtaining employee feedback on action plans (73% of respondents).
Although this evaluation’s review of public service surveys in other jurisdictions was not exhaustive, it suggested that public service employee surveys are widely used. Of particular note was the finding that all of Canada’s provinces and territories conduct public service employee surveys.
The PSES provides a valuable source of evidence for the Management Accountability Framework (MAF). MAF results reflect departmental and agency achievements and inform performance reviews of deputy heads. The Secretariat asks all departments and agencies to submit an organizational action plan based on their PSES results. According to Secretariat officials, MAF assessments are to include references to the organization’s self-assessment based on PSES results, along with related plans, actions and results. Although most interviewees were familiar with these requirements, few indicated that the PSES figured substantially in the MAF exercise. Correspondingly, about one third of survey respondents found the PSES useful in completing MAF assessments. That said, people management is one of seven areas of management assessed by MAF.
Alignment with government priorities and Secretariat roles and responsibilities
To what extent does the PSES program align with government priorities and with the Secretariat’s roles and responsibilities?
A wide range of documentation, including reports from the offices of the Clerk of the Privy Council and the President of the Treasury Board, as well as ministerial mandate letters, indicate the need to promote effectiveness, productivity and well-being within the public service. Most recently, Budget 2016 included a commitment to strengthen the public service, ensuring that it remains innovative, agile and high-performing, and that it should develop its leaders and ensure that workplace improvements are based on evidence obtained through employee surveys.
Performance: progress toward outcomes
Conclusion: The PSES offers valuable insight into employees’ opinions about their engagement on the job, the federal workforce and workplace, and leadership in the public service. It also helps identify concerns regarding these areas. However, more could be done to provide a deep understanding of these root causes, in particular those relating to leadership concerns. Federal executives across the Government of Canada regularly use PSES results in their departmental and agency plans for improving their people management practices. To a lesser extent, PSES results contribute to the Secretariat’s development of the government’s policies for people management. While the PSES has contributed to initiatives that aim to enhance public service employees’ engagement and performance, there is limited evidence of government-wide responses to address issues identified in the PSES. In parallel, while government-wide PSES results show positive change over time in some areas, there is also negative and little to no change over time in several areas.Footnote 4 The areas of performance management, a respectful workplace and an ethical workplace have become more positive over time, while results for job satisfaction, satisfaction with organization, career development, and organizational performance have become less positive over time. Little to no change has occurred in results on senior management, official languages, staffing and labour relations.
Achievement of expected immediate outcomes
To what extent has the PSES program contributed to increased insight into employees’ opinions on employee engagement, leadership, the workforce and the workplace?
Interviewees were virtually unanimous in stating that the PSES provided them with insight and understanding about employee engagement and workforce and workplace issues. Interviewees spoke of the following specific issues:
- work-life balance
- performance management
- career development
- empowerment
- communications
- employment equity
- telework and other flexible work arrangements
- harassment
- respect
- values and ethics
- discrimination
In all these areas, federal executives were able to identify areas of concern within their organization and within organizational units. They can compare their organization with others, and they can compare its current performance against its performance in previous years.
Interviewees were less positive about leadership, noting that the PSES reveals insights about leadership that are less precise and actionable than insights in other areas. They acknowledged that other means such as the APEXFootnote 5 survey, which has a different focus than that of the PSES, are used to gain insight.
In contrast, primary user survey respondents found the senior management leadership (73%) results to be most useful to their organizations. This suggests that while PSES results for leadership are valuable, there are expectations that more could be done to gather insight into this area.
In addition, interviewees noted that PSES findings are limited, as the PSES can reveal concerns but offers little to help in understanding their root causes. While interviewees acknowledged that further analysis and group discussions in their respective departments or agencies are necessary to understand these issues, some suggested that more specific questions that delve into root causes could be used in a future PSES or interim survey.
Consistent with the analysis of interviewee comments, 74% of primary user survey respondents reported that the PSES “contributed to providing insight into [my] department’s/agency’s employees’ opinions on employee engagement, leadership, the workforce and the workplace” to a “large extent” or a “very large extent.”
Other themes that survey respondents found to be most useful to their organizations are as follows:Footnote 6
- respectful workplace (72%)
- leadership: supervisor (69%)
- harassment (67%)
- employee engagement (65%)
- ethical workplace (64%)
- discrimination (61%)
Primary users of PSES information mainly consult the publicly available PSES website for current and previous surveys’ results. In terms of PSES products, between 72% and 86% of respondents reported finding usefulFootnote 7 departmental and agency results by question or theme through the PSES summary reports, the PSES highlights reports, and results by organizational structure. There was less uptake for GCconnex,Footnote 8 with 31% of respondents finding it useful.Footnote 9 Of the respondents who engaged in cross-government discussions, most cited GCconnex as the forum that they used. In addition, 20% of respondents noted that the Office of the Chief Human Resources Officer (OCHRO) should promote or improve the user experience on GCconnex to facilitate information sharing among departments and agencies.
Primary user survey respondents who participated in the PSES’s live presentation formatsFootnote 10 found them to be 70% to 87% “useful to at least a moderate extent in understanding and/or addressing [their organization’s] PSES results.” While 31% to 48% of respondents did not use or did not know how useful they were, only 8% of survey respondents were unaware, which suggests that communication of these presentations was sufficient. PSES post-fieldwork information sessions provided to PSES contacts were the most highly regarded, with 87% finding this mechanism useful to at least a moderate extent. PSES pre-fieldwork presentations to the Human Resources CouncilFootnote 11 were the second most highly regarded mechanism, at 78%.
Visits to the Secretariat’s PSES web pageFootnote 12 indicate frequent usage of the PSES reports posted there, but these statistics are not indicative of all usage since some departments and agencies post results on their own internal websites.
In all, 15 departments and agencies ordered customized reportsFootnote 13 from Statistics Canada.
Achievement of expected intermediate outcomes
To what extent has the PSES program contributed to: (a) department/agency plans developed to address issues and opportunities identified in the survey; and (b) changes to public service-wide people management policies?
Departmental and agency plans
Interviewees reported that PSES findings were a catalyst for planning, and most departments and agencies reported creating action plans. For example, in one department where there were concerns over information from senior management making its way to employees, “skip level” meetings were instituted to allow managers to be more attuned to what is happening at all levels. Other interviewees described the development of managerial practice guidelines that cover areas such as harassment, discrimination and inclusion, as well as taking steps to bring in guest speakers and arrange town hall discussions about employees’ concerns.
Consistent with findings from the analysis of interviewee comments, 60% of primary user survey respondents reported that the PSES contributed to their department or agency developing plans that address people management issues and opportunities.Footnote 14 In addition, 62% reported that the PSES contributed to developing or modifying people management policies according to issues of concern.Footnote 15
Treasury Board policies for people management
Secretariat policy centre managers who were interviewed reported that the PSES has not directly led to any changes in Treasury Board policies for people management. This observation is supported by the findings of this evaluation’s document review, as no evidence was found of a policy modification based on PSES results. That said, interviewees noted that the PSES revealed concerns regarding people management that provided key inputs into policy development. In addition to issuing policies, the Secretariat’s people management policy centres issue a range of related policy instruments such as guidelines and best practices for people management, and PSES results could have a more immediate impact on these kinds of instruments, according to interviewees.
Achievement of expected long-term outcomes
To what extent has the PSES program contributed to a high-performing and engaged public service?
Interviewees’ opinions varied on whether the PSES has contributed to improving public service employees’ engagement and performance. On the one hand, interviewees assumed that the knowledge gained from PSES results, and the plans that resulted from this knowledge, would lead to improvements. On the other hand, little in the way of tangible proof of improvements could be identified, and even where improvements were noted, it is difficult to definitively attribute them to the PSES.Footnote 16
The perceptions of interviewees were supported by primary user survey respondents:
- 55%Footnote 17 indicated that the PSES contributed to “supporting improvements leading to an engaged workforce in [my] department/agency”
- 48% indicated that the PSES contributed to “supporting improvements leading to a high-performing workforce in [my] department/agency”
- 40% indicated that the PSES contributed to “supporting improvements that lead to overall good governance and sound stewardship, enabling efficient and effective service to Canadians”
Notwithstanding these views, results that have improved are in the minority compared with those that have worsened or have had little change. Improvements of three or more percentage points from 2008 to 2014 at the government-wide level were noted for approximately 20% of the questions that span the three survey cycles, while approximately 40% of the same questions were less positive from 2008 to 2014 by three or more percentage points. That said, the government-wide level results may not show substantial changes that may occur in individual departments or agencies. The overall public service results are driven by six very large departments representing almost 50% of the population. In order to move the overall PSES results by more than a few percentage points, there needs to be a large shift in perceptions across very large to large organizations.
Comparisons between the 2014 results and the results for identical questions covered by the PSES in 2008 and 2011 are available on the 2014 PSES website. Comparisons can no longer be made with survey years prior to 2008 since a methodological change precludes them from being comparable.Footnote 18
Certain areas, specifically performance management, respectful workplace and ethical workplace, have improved over time. Upward trendsFootnote 19 are noted between 2008 and 2014 for the following specific questions:
- “I receive useful feedback from my immediate supervisor on my job performance”
- “My immediate supervisor keeps me informed about the issues affecting my work”
- “My immediate supervisor assesses my work against identified goals and objectives”
- “Overall, my department or agency treats me with respect”
- “I can complete my assigned workload during my regular working hours”
- “If I am faced with an ethical dilemma or a conflict between values in the workplace, I know where I can go for help in resolving the situation”
However, job satisfaction, satisfaction with the organization, career development, and organizational performance are areas that have been less positive over time. The following specific questions examined showed a declining trend over time:Footnote 20
- “I feel I can claim overtime compensation (in money or in leave) for the overtime hours that I work”
- “Overall, I like my job”
- “Essential information flows effectively from senior management to staff”
- “My department or agency does a good job of supporting employee career development”
- “I believe I have opportunities for promotion within my department or agency, given my education, skills and experience”
- “I am satisfied with my department or agency”
The following job challenges show an increasing trend:
- “I feel that the quality of my work suffers because of too many approval stages”
- “I feel that the quality of my work suffers because of having to do the same or more work, but with fewer resources”
Additionally, little to no change has occurred in results on senior management, official languages, staffing and labour relations.
A workplace that is intolerant of harassment is viewed as a fundamental right, as identified in the Canadian Human Rights Act.Footnote 21 Most interviewees indicated that addressing harassment is a priority and that they had analyzed and taken actions in their departments to address the issue. Furthermore, PSES questions have been modified in recent cycles to delve into the sources of harassment. Yet government-wide PSES results for harassment continue to remain relatively unchanged, with 19% of employees who indicated that they had been harassed.Footnote 22 In this evaluation’s review of other jurisdictions, the results for the UKFootnote 23 and for Australia,Footnote 24 on harassment and bullying in the workplace,Footnote 25 have also remained unchanged over several years despite their respective efforts.
More generally, the trends suggest that areas for improvement highlighted by the PSES over the years may not have been consistently or sufficiently acted upon. In particular, the evidence suggests that although positive actions and results may have taken place within individual departments and agencies, there has not been a government-wide response to problems of a sufficient magnitude that would reverse some negative trends.
There is limited evidence of government-wide responses to address issues identified in the PSES. While some interviewees questioned the need for more government-wide responses given their individual deputy head accountabilities, others suggested that OCHRO could better support departments and agencies by facilitating the sharing of strategies and best practices, as well as helping to effect culture change.
Most interviewees focused on issues in their own departments or agencies and had limited interaction with other departments to discuss PSES results. Only 28% of primary user survey respondents indicated that OCHRO-led cross-government discussions were sufficient.
Interviewees pointed to a level of cynicism about the PSES that may be attributed, at least in part, to this finding. Perhaps most telling were responses to the statement “I believe senior management will try to resolve concerns raised in this survey”; only 43% to 44% of public service employees agreed during the last three survey cycles.
Performance: design and delivery
Conclusion: The PSES is effectively designed and delivered, and there have been systematic efforts to continuously improve it. Notwithstanding this, there are four significant concerns regarding the PSES program:
- the frequency of the PSES’s iterations
- the time lag between fieldwork and the release of results
- limitations in the analysis of PSES data
- adequacy of government-wide support to address identified issues
To what extent is the design of the PSES operating as intended? To what extent has the PSES program identified program and survey tool improvements, including monitoring external best practices, to inform improvements?
Evidence from interviewees and survey respondents support the finding that the PSES, as managed by OCHRO and administered by Statistics Canada, is effectively run and is of a high standard. The 71.4% response rate is among the highest of the public service surveys of the jursidictions examined. Interviewees perceive Statistics Canada as trustworthy. The PSES is offered in a wide range of alternative formats, and few other public service surveys can make this claim. Interviewees indicate that the timing of the fieldwork period is well chosen, allowing the inclusion of seasonal workers, avoiding summer holidays, and producing results in time for the MAF exercise. Through the feedback that the Secretariat solicits from departments and agencies, the post-mortem exercise it conducts with Statistics Canada, and the contacts it maintains with public service survey managers in other jurisdictions, the Secretariat systematically identifies concerns and regularly improves the PSES program and its survey tools.
What concerns are there about the design and delivery of the PSES?
The most commonly mentioned concern in the interviews was the PSES’s relative infrequency. Some see the three years between iterations as being too long and as providing inadequate feedback, reducing the PSES’s credibility. Every other jurisdiction reviewed for the evaluation conducted its survey either every two years (most provinces in Canada) or annually (the UK, the US and Australia). However, 57% of survey respondents who had an opinion on this pointFootnote 26 reported that they did not want to see a change in the frequency of the PSES. Of the 43% who did want to see the frequency changed, roughly two thirds indicated a preference for a survey every two years. Respondents in both groups mentioned an interest in interim surveys conducted annually that focus on particular aspects.
Some of these issues are already being addressed by the Secretariat, with work underway, to design a new shorter government-wide annual survey to complement the existing PSES.
The second most commonly mentioned concern was about the time lag between the fieldwork period and the release of results. Most deputy heads, other interviewees and survey respondents saw a time lag of four to five months as problematic.
The sophistication of PSES data analysis was seen as inadequate. There are two aspects to this concern. First, the vast range of analysis possibilities offered by the PSES appears to go largely unexploited. For example, interviewees mentioned the potential value of comparing scores for similar workers across similar departments (for example, technicians in science-focused departments). A useful analysis might involve examining responses of teleworkers in a particular branch and region within an organization, and comparing responses of men and women. These kinds of analyses, however, are rarely done because of the following reasons:
- departments and agencies do not necessarily have the required research and analytical capacity
- the process of requesting such analyses from Statistics Canada is seen as cumbersome and expensive
- access to the raw data for non-Statistics Canada personnel is limited to academic researchers and to departments and agencies on a limited basis and under controlled conditions
Second, because the PSES tends not to include questions related to causal factors,Footnote 27 the only way to probe certain research questions more deeply is to collect more data. An example of such a research question is “What are the factors causing an unusually high number of employees in a particular branch and region to report that they plan to leave their jobs within the next two years?” Such data collection and research can be undertaken by departments and agencies only at their own expense. If follow-up research is to be conducted in departments and agencies, many follow-up questions of interest will go unanswered.
Some interviewees indicated that pre-survey consultations were not as comprehensive as some interviewees would have liked, saying that starting such consultations earlier would have been beneficial. In addition, the user-friendliness of PSES products was seen as somewhat lacking, with some suggestions made for using dashboards and infographics. Although it was acknowledged that the Secretariat is effective in modifying PSES content in line with current thinking, interviewees and survey respondents suggested that more could be done in this regard. Revised PSES content could examine the concerns of millennials, as well as more or different questions on the topics of civility, mental health, harassment, discrimination, innovation and flexible work arrangements. Accordingly, there is an appetite for greater government-wide support of more detailed analyses, strategies and tools that could facilitate departments and agencies to better understand their identified issues.
Performance: economy and efficiency
Conclusion: Although determining the PSES’s cost-effectiveness was beyond the scope of this evaluation, interviewees indicated that the PSES provides adequate value for money. Statistics Canada administers the PSES, although there may be alternatives from suppliers in the private sector that are more economical.
Is resource utilization at a reasonable level in relation to the production of PSES program outputs and outcomes?
It is difficult to empirically determine the cost-effectiveness of the PSES. Direct expenditures associated with the PSES are neither the highest nor the lowest on a per-respondent basis among the public service surveys examined among other jurisdictions. At approximately $2.3 million, OCHRO’s direct expenditures represent a small portion of the total cost of the PSES.Footnote 28 Interviewees, however, assumed that OCHRO worked to ensure that expenditures were economical, and indicated that it was important to conduct the PSES, irrespective of its cost.
Are there more affordable alternatives to producing PSES outputs and outcomes?
Most jurisdictions examined used their own internal statistical agency to conduct their surveys. Some, however, contracted their surveys to private firms. In some of Canada’s provinces, Ipsos, TalentMap and the Hay Group are used. The UK and Australia use ORC International.Footnote 29 ORC’s price of $1,005,000Footnote 30 to the UK government was found to be the most economical among jurisdictions examined. In comparison, the Secretariat paid $1,201,240 to Statistics Canada to administer the 2014 PSES.
Conclusions and recommendations
Conclusions
The evidence shows the continuing need for the PSES as a reliable and comprehensive tool to periodically determine opinions from public service employees about their jobs and their workplace. The PSES addresses government priorities regarding its support for an effective public service, and administration of the PSES fits within the Secretariat’s mandate.
The PSES offers valuable insight into employees’ opinions about their engagement on the job, the federal workforce and workplace, and leadership in the public service. It also helps identify concerns regarding these areas. However, more could be done to provide a deep understanding of the root causes of these concerns, in particular those relating to leadership. Federal executives across the Government of Canada regularly use PSES results in their departmental and agency plans for improving their people management practices. To a lesser extent, PSES results contribute to the Secretariat’s development of the government’s overall tools for people management.
The PSES has contributed to initiatives that aim to enhance public service employees’ engagement and performance. However, while government-wide PSES results show positive change over time for some areas, there is also negative and little to no change over time in several areas.
The areas of performance management, a respectful workplace and an ethical workplace have improved over time, while results for job satisfaction, satisfaction with organization, career development and organizational performance have weakened over time. Little to no change has occurred in results on senior management, official languages, staffing and labour relations.
The PSES is effectively designed and administered, and there have been systematic efforts to continuously improve it.
Notwithstanding this, there are four significant concerns regarding the PSES program:
- the frequency of the PSES’s iterations
- the time lag between fieldwork and the release of results
- limitations in the analysis of PSES data
- adequacy of government-wide support to address identified issues
Although determining the PSES’s cost-effectiveness was beyond the scope of this evaluation, interviewees indicated that the PSES provides adequate value for money. Statistics Canada administers the PSES, although there may be alternatives from suppliers in the private sector that are more economical.
Recommendations
- The Secretariat should develop a strategy to proactively address government-wide issues highlighted in the PSES results. Such a strategy should include the Secretariat:
- playing a stronger leadership role in government-wide initiatives for change while respecting individual deputy head accountabilities
- bringing greater consequences to departments and agencies regarding their PSES performance, depending on their level of improvement
- taking specific actions to address those issues that could benefit from government-wide intervention
- The Secretariat should develop a plan to enhance the PSES program that addresses the following:
- providing timelier insights to departments and agencies on employee engagement, the workplace, the workforce and leadership issues
- facilitating ongoing consultation with organizations and continual improvement of survey tools and products
- enabling a wider range of analysis, including collecting additional targeted data
- ensuring that the PSES represents the state of the art in survey methodology and remains pertinent to key user and stakeholder groups
Appendix A: the PSES 1999 through 2014
The survey
The PSES gauges the opinions of Canadian federal public service employees about their jobs and their workplace. The PSES is administered online, with alternative formats available to accommodate employees with disabilities and employees without computer access. It is sent to employees of organizations in the core public administration and of participating separate agencies listed in Schedules I, IV and V of the Financial Administration Act. In 2014, the PSES was launched on August 25 and included over 250,000 public service employees in 93 departments and agencies. Over the years, it has elicited a high response rate; in 2014, a total of 182,165 employees responded, representing 71.4% of the target population.
Modifications to the PSES have been made with each iteration to improve it and to keep up to date on emerging issues. The 2014 PSESFootnote 31 contained 106 questions on a wide range of issues, including public service employees’ jobs, their organizations and how employees are managed, as well as demographic questions.
PSES results are made available to the federal public service and to the public through the Secretariat’s website. Results are available at various levels, such as the public service as a whole, by organization, by units within organizations and by demographic characteristics such as age, gender, first official language, part-time work and full-time work.
The PSES’s governance, roles and process
The PSES was managed by the Secretariat in 1999 and 2002, the Canada Public Service Agency in 2005 and 2008, and the Secretariat’s Office of the Chief Human Resources Officer (OCHRO) in 2011 and 2014. OCHRO is responsible for PSES planning, content, pre-survey consultations and report templates (75 templates were created for the 2014 PSES), and for presenting findings and other follow-up activities. OCHRO stays abreast of best survey practices, including being a member of an interjurisdictional team with other public service survey counterparts in Canada and maintaining contact with counterparts elsewhere. OCHRO’s work on the PSES is supported by the Secretariat’s internal Communications unit and its Web Services unit.
Since the PSES’s inception, Statistics Canada has carried out technical preparation and fieldwork under contract to the Secretariat. Statistics Canada is responsible for creating the sampling frame (gathering employee email addresses), determining organizational units in consultation with departments and agencies, focus group testing of new and revised questions, and administering the PSES, including providing helpdesk support, developing the weighted dataset, and producing aggregated datasets and reports.
Responsibility for the PSES rests with the Chief Human Resources Officer and the Assistant Deputy Minister, Governance Policy and Planning (GPP) Sector, of the Secretariat. The Secretariat’s PSES team is led by the Director of Performance, Measurement and Monitoring, and is supported by the Director, Special Surveys Division of Statistics Canada. Guidance to the team is provided by a steering committee co-chaired by the Executive Director, Strategic Infrastructure and Information Management, OCHRO, and the Director General, Health, Justice and Social Surveys, Statistics Canada. Overall governance is provided by a board that provides direction to the steering committee and that resolves issues that cannot be resolved at other levels of governance.
Pre-survey
In order to update PSES content, prior to the fieldwork for the 2014 PSES, the Secretariat consulted internally with its policy centres, including those responsible for employment equity and diversity, values and ethics, and official languages. The Secretariat consulted externally with employment equity champions and chairpersons of committees, central agencies, departments and agencies, and bargaining agents. For the 2014 PSES, departments and agencies were offered the opportunity to add up to five supplementary questions for their organizations; 12 organizations took advantage of this opportunity.
Over the history of the PSES, the priorities of the Clerk of the Privy Council and the President of the Treasury Board have been taken into account. In addition, academic research has been consulted.Footnote 32 Content changes for the 2014 PSES as a result of these efforts included the addition of questions on the duty to accommodate, types of harassment, and actions taken following experiences of harassment or discrimination. Questions were also added to explore performance management, given the launch of the Treasury Board’s Directive on Performance Management in .
Post-survey
Following the administration of the PSES, Statistics Canada processes data, populates reports and tables, and produces and disseminates aggregate datasets. Products for the 2014 PSES included the following:
- results tables for the overall public service and for departments and agencies, including breakdowns by demographic characteristics and breakdowns by organizational structure
- open datasets, which are results in a spreadsheet format to allow users to create their own tablesFootnote 33
- 2014 PSES Summary Reports for the overall public service and individual organizations
- Focus Series reports, which featured 14 short reports on various themes such as engagement and respect
Because PSES findings are publicly available, anyone inside or outside the public service can make use of them. However, the primary target audience of the PSES comprises the following:
- departmental and agency deputy ministers and executives
- departmental and agency heads of human resources
- designated PSES champions and others who have assigned duties related to the PSES (this group constitutes the Secretariat’s “PSES contacts” list)
- heads of Treasury Board policy centres for people management
PSES products in 2014 were made available in two waves. On , four months after the fieldwork closed, “top-line”Footnote 34 results and results broken down by demographic characteristics for the overall public service and individual departments and agencies were released. On , the remaining products were released, which included breakdowns by organizational unit. The release of PSES findings was accompanied by Secretariat presentations to departmental and agency heads of human resources and other audiences.
By request, and on a cost-recovery basis, Statistics Canada produces summary reports according to an organization’s specifications (for example, by sector or branch, occupational group, or employment equity group) and cross-tabulations of results (for example, by gender and age).
Following the release of PSES results, the Secretariat seeks feedback from departments and agencies about the survey process and products. Following the 2014 PSES, 38 departments and agencies provided feedback on the PSES’s content and processes, including ideas for improvement. In cooperation with Statistics Canada, the Secretariat also conducts its own post-mortem analysis. Observations from these exercises are used to improve the next iteration of the PSES.
Resources
The Secretariat’s work on the 2014 PSES began in and continued until . Payments to Statistics Canada were made over three fiscal years, starting in the 2013 to 2014 fiscal year and ending in the 2015 to 2016 fiscal year. The Secretary’s Reserve Fund provided the funds for these payments. Table 1 shows total resources expended by the Secretariat on the 2014 PSES. The total of $2,338,022 does not include costs to departments and agencies, such as salaries and other costs.
| 2012 to 2013 | 2013 to 2014 | 2014 to 2015 | 2015 to 2016 | Total | |
|---|---|---|---|---|---|
| Full-time full-year equivalentsFootnote 35 | 1.67 | 4.63 | 3.31 | 2.09 | 11.70 |
| Salaries | $162,470 | $464,910 | $317,652 | $191,750 | $1,136,782 |
| Statistics Canada invoices | $25,000 | $1,105,000 | $71,240 | $1,201,240 | |
| Total costs | $162,470 | $489,910 | $1,422,652 | $262,990 | $2,338,022 |
Appendix B: PSES logic model
Appendix B - Text version
The main components of the Public Service Employee Survey program are:
- activities
- outputs
- immediate outcomes
- intermediate outcomes
- long-term outcome
- TBS strategic outcome
The program’s activities are to manage the survey via:
- project management, communications and monitoring
- pre-survey preparation
- data collection
- release of results
The program outputs are:
- communication and information tools, a letter of agreement and a survey questionnaire
- survey results from across the public service (reports); department and agency survey results (reports); and research, analysis and advice
- evaluation, post-mortem reviews and feedback sessions
The immediate program outcomes are:
- insight into public service employees’ opinions on employee engagement, leadership, the workforce and the workplace
- insight into departments’ and agencies’ opinions on employee engagement, leadership, the workforce and the workplace
The intermediate program outcomes are:
- current and accurate information available to inform people management policies across the public service
- plans developed that address issues and opportunities identified through employee surveys
The long-term outcome of the program is that improvements lead to a high-performing and engaged public service that delivers on government priorities now and in the future.
The TBS strategic outcome is good governance and sound stewardship to enable efficient and effective service to Canadians.
Appendix C: evaluation issues, questions and indicators
Relevance
Issue 1: continued need for program
Assessment of the extent to which the PSES program continues to address a demonstrable need and is responsive to the needs of Canadians
- To what extent is the PSES meeting current and emerging needs with respect to assessing the state of the workforce, workplace and leadership?
Issue 2: alignment with government priorities
Assessment of the linkages between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes
- To what extent does the PSES program align with federal government and department/agency priorities?
Issue 3: alignment with federal roles and responsibilities
Assessment of the role and responsibilities of the federal government in delivering the program
- To what extent does the PSES program align with the Secretariat’s roles, responsibilities and priorities?
Performance (effectiveness, efficiency and economy)
Issue 4: achievement of expected outcomes
Assessment of progress toward expected outcomes (including immediate, intermediate and ultimate outcomes), with reference to performance targets and program reach, program design, including the linkage and contribution of outputs to outcomes:
- To what extent has the PSES program contributed toward increased insight into public service employees’ opinions on employee engagement, leadership, the workforce and the workplace?
- To what extent has the PSES program contributed to progress toward increased insight into departmental and agency employees’ opinions on employee engagement, leadership, the workforce and the workplace?
- To what extent has the PSES program contributed toward current and accurate information available to inform people management policies across the public service?
- To what extent has the PSES program contributed toward plans that are developed that address issues and opportunities identified through employee surveys?
- To what extent has the PSES program contributed to improvements toward a high-performing and engaged public service that delivers on government priorities now and in the future?
- To what extent is the design of the PSES operating as intended?
- To what extent has the PSES program identified program and survey tool improvements, and associated plans and actions to improve the PSES?
- To what extent does the PSES program monitor external best practices to inform PSES program strategies and tools?
Issue 5: demonstration of efficiency and economy
Assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes
- Is resource utilization at a reasonable level in relation to the production of PSES program outputs and outcomes?
- Are there more affordable alternatives to producing PSES outputs and outcomes?
Appendix D: methodology
This evaluation of the PSES program is based on several sources of evidence, as indicated in the following.
Review and analysis of documentation and administrative data
In order to understand the PSES program’s context, and to identify and analyze specific performance indicators, 41 documents were reviewed. These included the following:
- foundational documents such as the project charter and the letter of agreement with Statistics Canada
- a sample of departmental and agency Management Accountability Framework (MAF) reports
- relevant central agency reports
- PSES activity reports
In addition, rudimentary trend analyses were undertaken using PSES results from 1999 onward. The 1999 to 2005 and 2008 to 2014 survey cycles were analyzed separately because the response scale changed from a four-point scale to a five-point scale starting in the 2008 survey cycle. Descriptions of each label on the scale also changed. With these variables in play, while legitimate comparisons can be made between the 1999 to 2005 survey cycle results, and between the 2008 to 2014 survey cycle results, any comparisons between these two sets of results would not be methodologically sound.
Jurisdictional comparison
To provide additional context, and especially to determine best practices with a view to supporting suggestions for improvement, the public service surveys of all of Canada’s provinces and territories, as well as the US, the UK and Australia, were benchmarked. The benchmarking process included conducting a review of available documentation and interviews with survey officials from each jurisdiction.
Key informant interviews
Primary data pertaining mainly to performance questions were obtained through interviews with the following:
- 9 deputy heads of departments and agencies that participated in the 2014 PSES
- 1 representative of a non-participating agency
- 10 Secretariat executives
- 9 stakeholders representing Statistics Canada, central agencies, the Human Resources Council and bargaining agents
User survey
Primary data were also obtained more widely through a survey of departmental and agency officials who are responsible for the PSES in their organizations. The Secretariat’s PSES team provided the names and coordinates of key departmental and agency contacts to the evaluators. Online survey questionnaires that focused on performance, design, delivery and alternative questions were sent to 228 officials, with 166 responses received, representing a 72.8% response rate:
- 87% of respondents were either departmental or agency PSES champions (31%) or supported their organization’s champion (56%)
- 46% were heads or managers of human resources functions
- 10% were deputy ministers
- 10% were assistant deputy ministers
Of participating departments and agencies, 83% are represented in the respondent sample.
Limitations
Although the methodology used was comprehensive and deemed to be adequate for the requirements of the evaluation, it had limitations. Other than the views of union representatives, employee perspectives are not represented in the study. Deputy heads of non-participating departments and agencies are represented only by an interviewee from one small agency. Jurisdictions approached were reluctant to release details on expenditures, and only three international jurisdictions were examined. Changes to response scales (described above) limited the trend analysis that could be performed on PSES historical results. To the extent possible, these limitations were mitigated by the use of evidence from other available sources.
Management Response and Action Plan
Evaluation of the Public Service Employee Survey
The Governance, Planning and Policy (GPP) Sector of the Office of the Chief Human Resources Officer (OCHRO) has reviewed the evaluation of the 2014 Public Service Employee Survey (PSES) program.
Recommendation 1
The Secretariat should develop a strategy to proactively address government-wide issues highlighted in the PSES results. Such a strategy should include the Secretariat:
- playing a stronger leadership role in government-wide initiatives for change while respecting individual deputy head accountabilities
- bringing greater consequences to departments and agencies regarding their PSES performance, depending on their level of improvement
- taking specific actions to address those issues that could benefit from government-wide intervention
OCHRO fully agrees with the first two bullets of the recommendation. OCHRO agrees in principle with the third bullet, although related activities are limited by TBS’ mandate.
| Proposed action | Start date | Target completion date | Office of primary interest |
|---|---|---|---|
|
|||
|
(presentation to Public Service Management Advisory Committee for consideration) Full implementation in time to inform the 2018 annual survey cycle |
GPP/OCHRO | |
|
|||
|
GPP/OCHRO | ||
|
Ongoing | GPP/OCHRO | |
|
Ongoing | TBD | |
|
|||
|
GPP/OCHRO | ||
|
Ongoing | GPP/OCHRO | |
Recommendation 2
The Secretariat should develop a plan to enhance the PSES program that addresses the following:
- providing timelier insights to departments and agencies on employee engagement, the workplace, the workforce and leadership issues
- facilitating ongoing consultation with organizations and continual improvement of survey tools and products
- enabling a wider range of analysis, including collecting additional targeted data
- ensuring that the PSES represents the state of the art in survey methodology and remains pertinent to key user and stakeholder groups
GPP agrees with all aspects of recommendation 2.
| Proposed action | Start date | Target completion date | Office of primary interest |
|---|---|---|---|
|
Results: |
Ongoing | Deputy heads |
|
Summer 2017 | GPP/OCHRO | |
|
Spring 2017 | Ongoing | GPP/OCHRO |
|
|||
|
Ongoing | GPP/OCHRO Departments and agencies |
|
|
|||
|
Spring 2017 and ongoing | Ongoing | GPP/OCHRO |
Report a problem or mistake on this page
- Date modified: