Evaluation of Employee Assistance Services 2009-2010 to 2013-2014

Prepared by Office of Evaluation Health Canada and the Public Health Agency of Canada

August 2014

Table of Contents

Executive Summary

Evaluation Purpose and Scope

The purpose of the evaluation was to assess the relevance and performance of the Employee Assistance Services (EAS) for the period of 2009-2010 to 2013-2014.

The evaluation was required by the Treasury Board of Canada's Policy on Evaluation (2009), which now requires that all direct program spending be evaluated every five years.

Program Description

The EAS, part of the Regions and Programs Bureau (RAPB) of Health Canada, is currently the service provider of employee assistance services for approximately 140 federal organizations1. EAS clientele is comprised of the employees (and their dependents) from federal organizations under agreement with Health Canada. The overall EAS clientele is currently estimated at 1.6 million Canadians.

EAS supports the Government of Canada in fulfilling its obligation to protect the health and safety of its employees through the delivery of four EAS components.

  1. Employee Assistance Program (EAP) is the component generating the most activity. This component provides primarily professional counselling and advisory services (e.g., soft skills/advice/coaching) to employees.
  2. Specialized Organizational Services (SOS) provide customized training sessions and workshops to employees as well as long-term coaching to managers and executives.
  3. Occupational Critical Incident Stress Management (OCISM) Services support the Government of Canada in fulfilling its obligation to protect the health and safety of the 1,000 nurses working in remote and/or isolated First Nations communities.
  4. Psycho-social Emergency Preparedness and Response (PSEPR) Services provides support to employees impacted by or deployed to any events (e.g., earthquake) requiring an emergency-type of response. PSEPR is the only component funded through A-Base funding.

EAS operates on a cost recovery basis according to a vote-net revenue (VNR)2 authority approved by Treasury Board. EAS delivery relies on 50 employees located mostly in the National Capital Region as well as on a network of more than 800 contracted mental health professionals, who are referred to as affiliate providers.

Conclusions - Relevance

Based on the information reviewed over the course of EAS evaluation, it can be concluded that there is a demonstrated need for interventions such those delivered by EAS and that it is the role of the Government of Canada, and especially of Health Canada, to protect the health and safety of federal employees.

All lines of evidence confirmed that mental health issues, such as anxiety, stress and depression, regardless of where or how they arise, can easily cross back and forth between the workplace and family life and impact on the wellbeing and productivity of individuals.

Considering the federal government, and specifically Health Canada, has also clear roles and responsibilities with respect to promoting and preserving the health (including the mental health) of its employees, as established in the: Department of Health Act; Treasury Board Policy on Occupational Safety and Health; Treasury Board Policy on Employee Assistance Program; as well as in various collective agreements, it is evident that there is an ongoing need for some form of intervention to mitigate the impact of stress associated with life or work demands.

The Workplace Wellness and Productivity Strategy (2014) developed by the Treasury Board is an obvious government priority aimed at ensuring federal employees receive the services necessary to minimize absenteeism and long term disability.

Conclusions - Performance

Achievement of Expected Outcomes (Effectiveness)

EAP and OCISM seem to contribute to the increased use of resiliency skills and tools and thus contribute to increased productivity for the users remaining at work and to a modest reduction in absenteeism.  Data is not sufficient to conclude on the extent to which SOS' and PSEPR contributed to attitude or behaviour change.

There is evidence indicating that EAS increased its client base significantly over the course of the evaluation period. The increases in the proportion of calls to Crisis and Referral Center (CRC) were significantly lower than the increase in the consortium employee population suggesting there may be an awareness gap in departments having recently joined the EAS consortium or that the department joined the consortium late in the fiscal year.

EAP and OCISM seem to contribute to the increased use of resiliency skills and tools to help federal employees in finding satisfaction at work. Data available also suggest that EAP contributes to increased users' productivity for those remaining at work and contributes to a reduction in absenteeism. With respect to other EAS service lines, there is not sufficient data to conclude as to whether those services contribute to attitude or behaviour change after the intervention.

Demonstration of Economy and Efficiency

EAS is delivered in an economic and efficient manner.

For the most part, EAS does demonstrate efficiency and economy. EAS does deliver high quality services in accordance with industry and professional standards and has mechanisms in place to ensure economical use of public funds although updated technological infrastructures could contribute to greater efficiencies, e.g., for the collection of information and compensation of affiliate providers. When compared to one other jurisdiction, it appears that EAP fee per employee is in line with this comparator.

Recommendations

The evaluation identified the following recommendations:

Recommendation 1

EAS examine its role within the Treasury Board Workplace Wellness and Productivity Strategy to identify measures contributing to the prevention of issues potentially impacting mental health of employees.

Given the broad thrust of the Strategy, EAP may need to develop additional services and/or tools and enhance existing ones to further support the prevention, education and awareness objectives of the Strategy, such as health risk assessment and / or further promoting e-counselling. EAS should explore how SOS can more significantly contribute to support the objectives of the Strategy.

Recommendation 2

EAS examine what data should be collected and analyzed to develop a more fulsome analysis of the needs and impact of EAS on productivity of employees.

These analysis should be used by EAS to adapt its service offer and to inform senior management of client departments. This could include:

  • reviewing the management information and performance measurement systems to ensure data support the objectives of the Workplace Wellness and Productivity Strategy;
  • considering the standards developed by the Mental Health Commission of Canada as a possible starting point for EAS to refine its datasets; and
  • developing partnerships in client-departments that would allow EAS to communicate the results of their analysis to senior departmental representatives to inform their decision making.

Management Response and Action Plan

Employee Assistance Services, Specialized Health Services Directorate, Regions and Programs Branch
Recommendations Response Action Plan Deliverables Expected Completion Date Accountability Resources

EAS examines its role within the Treasury Board Secretariat's Workplace Wellness and Productivity Strategy (WPS) to identify measures contributing to the prevention of issues potentially impacting mental health of employees.

EAS examines what data should be collected and analysed to develop a more fulsome analysis of the needs and impact of EAS on productivity of employees.

Agreed - EAS is following the developments related to WPS initiative in order to align and deliver support services to Federal Public Servants.

Agreed - EAS has documented measures for enhanced data collection and analysis.

EAS will work collaboratively with Health Canada's project lead for Workplace Wellness and Productivity Strategy.

Formal TBS recommendations have not been communicated at this time.

EAS has joined PWGSC's Shared Case Management System initiative, and is developing a new IT / business tool / database that will increase efficiency, and improve data collection and analysis.

EAS is undertaking some expansion and enhancement of current offerings (expanding the scope of EAP services, promoting and developing resources for implementing the National Standard for Psychological Health and Safety in the Workplace).

EAS is committed to raising awareness of Mental Health and associated services available through SOS to support the healthy workplaces.

  • Increase capacity to deliver Mental Health First Aid training for departments looking to implement the standard.
  • Support departments in various stages of implementation (e.g. analysing pre and post evaluations, establishing next steps, reporting against the Standard) through Specialized Organizational Services (SOS).
  • EAS is developing a joint project with TBS and the CSPS to deliver one-day training for managers on "Managing Mental Health in the Workplace".

Shared Case Management System - New IT business tool / database for EAS.

Data improvements include real time utilization reports, immediate client feedback, and capturing additional information from clients at intake, leading to improved outcome measurement. EAS will continue to use the outcome measurement questions developed as part of this evaluation.

Should Departments express interest, compiling data from various sources (LR, HR, Ombudsman, along with EAP data) to produce more detailed reports for departments.

As of August 2014, over 20 providers are already on Standing Offer Agreement to deliver MHFA (through SOS).

Recruit additional professionals with knowledge and expertise related to the Standard - from September 1, 2014 to June 2015.

Pilot of first course - target January 2015. If the pilot is successful, EAS will advocate to make this training essential for managers.

IP project 125 is targeting gate 3 approval in the fall of 2014.

As part of the Shared Case Management initiative, some aspects of the project will be outside of our control.

EAS estimates that the new system will be fully operational by April 2016.

Executive Director, Specialized Health Services Directorate; and

Senior Director General, Regions and Programs Bureau

Executive Director, Specialized Health Services Directorate; and

Senior Director General, Regions and Programs Branch

These deliverables will be developed and resourced on a cost recovery basis, as per EAS' current funding model.

These deliverables will be developed and resourced on a cost recovery basis, as per EAS' current funding model.

Costs and resources for this initiative are being managed through the Health Canada Investment Plan (IP). With approval, resources will be allocated from IMSD, EAS, BAID, and RAPB's innovation reserve.

1.0 Evaluation Purpose

The purpose of the evaluation was to assess the relevance and performance of the EAS for the period of 2009-2010 to 2013-2014.

The evaluation was also required by the Treasury Board of Canada's Policy on Evaluation (2009), which now requires that all direct program spending be evaluated every five years.

2.0 Program Description

2.1 Program Context

According to the Mental Health Commission of Canada,3 one in five Canadians will experience a mental health problem or illness in any given year with nearly a quarter of the country's working population currently affected by mental health problems or illnesses leading to absenteeism, "presenteeism" and turnover. Approximately 30 per cent of short- and long-term disability claims in Canada are attributed to mental health problems and illnesses with adults in their early and prime working years being among the most vulnerable.

The Treasury Board policy on Employee Assistance Program4 currently in place states that departments must ensure that employees are provided with services that conform to the policy, with the objective to foster and maintain the employees' well-being and productivity. Services to be provided should be confidential and provided by practitioners who abide by the Code of Ethics included in the policy.

While federal departments can select a provider of their choosing to comply with the policy, EAS, a division of the Specialized Health Services Directorate (SHSD), in the RAPB of Health Canada; is currently the service provider of employee assistance services for approximately 140 federal organizations5. The remaining federal organizations under agreement with other service providers are: the Canada Revenue Agency (Ottawa headquarters), the Canadian Food Inspection Agency, Public Works and Government Services Canada, Defence Construction Canada, Natural Resources Canada, Office of the Superintendent of Financial Institutions Canada, Canadian Museum for Human Rights, Foreign Affairs and International Trade Canada, and Canada Border Services Agency.

EAS clientele is comprised of the employees (and their dependents) from federal organizations under agreement with Health Canada, including the civil employees of the Department of National Defense as well as active and retired members of the Canadian Armed Forces and the Royal Canadian Mounted Police (RCMP). The overall EAS clientele is currently estimated at 1.6 million Canadians.

EAS operates on a cost recovery basis according to a VNR6 authority approved by Treasury Board. Treasury Board increased the VNR authority several times since 2001. The EAS VNR authority was at $8M for the period covered by the evaluation.

EAS operates according to the standards and procedures established by the Council on Accreditation (COA) -- an international, independent accrediting body that promotes the best known practices in the industry. EAS delivery relies on 50 employees located mostly in the National Capital Region as well as on a network of more than 800 contracted mental health professionals, who are referred to as affiliate providers.

This 2014 evaluation is the first formal evaluation of EAS.

2.2 Program profile

As one of the service providers on the market, EAS supports the Government of Canada in fulfilling its obligation to protect the health and safety of its employees through the delivery of four EAS components.

  1. Employee Assistance Program (EAP) is the component generating the most activity. This component provides:
    • Up to eight hours of voluntary and confidential professional counselling to assist employees and their dependents;
    • Advisory services (e.g., soft skills/advice/coaching) to employees working in a supervisor capacity, and,
    • Support to mitigate the long-term effects in the event of the traumatic incident, e.g., a colleague having a heart attack in the workplace.
  2. Specialized Organizational Services (SOS) provide customized training sessions and workshops to employees as well as long-term coaching to managers and executives. The workshops and coaching offered by SOS are provided on a fee for service basis and thus require management pre-approval.
  3. Occupational Critical Incident Stress Management (OCISM) Services support the Government of Canada in fulfilling its obligation to protect the health and safety of the 1,000 nurses working in remote and/or isolated First Nations communities. Services are primarily targeted to nurses hired by Health Canada. Band employed nurses would not be eligible to services in normal circumstances, although exceptions can happen. OCISM services are funded through a Memorandum of Understanding with the First Nations and Inuit Health Branch (FNIHB).
  4. Psycho-social Emergency Preparedness and Response (PSEPR) Services were started in the wake of the 9/11 attacks as part of a larger government emergency preparedness response to terrorist attacks, natural disasters and major accidents.  As part of its role with respect to emergency preparedness and response, Health Canada has been designated as the federal government provider of psychological and emotional support services, contributing to fulfill the Government of Canada obligation to protect the health and safety of its employees impacted by or deployed to any of these events (e.g., earthquake). PSEPR is the only component funded through A-Base funding.

Service Delivery

Annual interdepartmental letters of agreement between Health Canada and client-departments govern the delivery of EAS. Although there may be departmental particularities, these agreements define the eligible population including the concept of family/dependents, describe the services to be provided under the agreement and at what cost. For EAP, a fixed fee per employee is charged to the organization up to an eight per cent utilization rate. The extra fees that apply when the eight per cent threshold is exceeded are also described in the letter of agreement. The annual amount charged to the client-department for the delivery of EAP is recovered quarterly. The exceptions to this arrangement are the Department of National Defence (DND) and Veterans Affairs Canada (VAC), given these organizations' mandate, size of eligible employee population and a different usage pattern. These two departments are billed according to the service used rather than being charged a fixed fee per employee.

Services offered through SOS are billed to client-departments. This arrangement presents client-departments with the opportunity to hire EAS for the delivery of SOS as needed, although departments also have the choice to engage with another provider. Upon requests from a client-department, a small contract with an affiliate provider would be put in place for the delivery of a workshop or another activity, as appropriate.

For EAP, services are delivered by a network of 800 contracted mental health professionals located across the country and who are referred to as affiliate providers.  Federal employees who need the service can call the CRC 24 hours a day where the call will be answered by an intake worker. Intake workers have a Masters degree with a minimum of 5 years of experience. If the caller is experiencing a crisis, the intake worker will provide immediate counseling as necessary. For other cases, the client is called back within 24 hours to set up a first meeting to make an initial assessment. Before the end of the 3rd session, affiliate providers must submit an intervention plan to EAS management. In exceptional circumstances where community resources are not readily available, affiliate providers may be authorized to go beyond 8 sessions.

A subset of affiliate providers is available to provide services under SOS and PSEPR. For PSEPR, a subset of affiliate providers in all regions do prevention work, raising management awareness of the signs they need to be able to identify in employees providing services in an emergency situation. In the occurrence of such event, this subset of affiliate providers who are available 24 hours, 7 days a week, can be deployed. OCISM service delivery relies mostly on nurses, all of whom are located in Winnipeg.

2.3 Program Logic Model and Narrative

The program's logic model was developed by the Office of Evaluation in collaboration with EAS for the purpose of the 2014 evaluation.  The logic model is presented in Appendix 2.

As noted earlier, EAS supports the Government of Canada's obligation to protect the health and safety of its employees and their dependents through the four components described in Section 2.2 (Program Profile).

EAS underlying theory suggests that individuals involved with any of the four components will become more aware of his/her own strengths and resiliency skills. By using these skills more frequently in various aspects of life, EAS users are anticipated to improve coping with stressful events. Stress may lead to anxiety and unproductive behaviours that influence job satisfaction. With improved stress management skills, individuals exposed to the program should be in better position to detach themselves from stressful circumstances and maintain or increase their overall job satisfaction, which should translate into a reduced need to stay away from work (reduced absenteeism) and increased focus while at work (reduced presenteeism).

The EAP is targeted to employees of those federal departments who currently have an agreement with Health Canada. In addition, employees' dependants are also eligible to EAP services.

2.4 Program Alignment and Resources

Health Canada's 'Specialized Health Services' (Component 1.2) contributes to Strategic Outcome 1 of Health Canada's Program Alignment Architecture 2014-2015, 'A Health System Responsive to the Needs of Canadians'.

A financial overview of EAS for the period 2009-2010 to 2013-2014 is provided in Table 1. As noted earlier, EAS operates on a cost recovery basis according to a VNR authority approved by Treasury Board which means that there is no precise annual funding amount allocated to SHSD for the delivery of EAS but rather a maximum of annual revenue ($8M for the period covered by the evaluation) that can be generated to fund activities. According to the SHSD, a range of 20% under or above this authority is acceptable, which explains why revenues presented in Table 1 exceed the $8M authority.

Table 1: Overview of EAS Revenues
Categories 2009-10 2010-11 2011-12 2012-13 2013-14
EAP $4,834,512 $5,742,304 $5,807,944 $6,416,360 $6,408,911
Revenues from billed departments $1,965,234 $2,071,307 $2,284,846 $3,126,166 $2,482,981
SOS $1,257,000 $1,226,916 $1,801,228 $3,217,500 $1,798,000
OCISM $1,143,411 $1,055,524 $968,200 $1,42000 $1,420,000
PSEPR $129,740 $129,740 $129,740 $129,740 $129,740
EAS Promotions N/A $9,173 $9,357 $17,390 $7,520
Total $9,329,897 $10,234,964 $11,013,315 $14,327,156 $12,247,152
Source: Employee Assistance Services

* The RCMP, DND and VAC are either billed or have not been continuously part of the consortium during the evaluation period.

Revenues obtained from the RCMP, DND and VAC have been included in this table to provide the reader with full information of EAS revenue. However, these departments' use of EAS services is excluded from the scope of this evaluation (see section 3.1 for additional detail) given they have not been consistently part of the consortium during the evaluation period.  An analysis of financial information above, excluding information from billed departments, suggests two elements likely contributed to increase EAS revenues: the growth in the consortium population (resulting from additional departments entering into agreement with Health Canada) and the cuts announced in the 2011 Budget. According to SHSD, both of these elements impacted EAP and SOS use, thus increasing EAS revenues. The decline in EAS overall revenue that is reported for 2013-2014 is largely associated with revenues stemming from SOS returning to their normal level. SOS is the component that had increased the most following the 2011 Budget announcement.

Revenues stemming from promotional activities refer to revenue generated through the sale of promotional material to client-departments, in addition to the promotional material already provided under the letter of agreement between the client-department and Health Canada. This represents a small portion of EAS revenues which peaked in the year following the 2011 Budget announcement, where most client-departments were trying to support their staff.

3.0 Evaluation Description

3.1 Evaluation Scope, Approach and Design

Each year, in collaboration with the program, an evaluation risk is assigned to each program in the Health Canada-Public Health Agency of Canada 5-year Evaluation Plan. This evaluation risk, which is based on a number of factors including program complexity and maturity, materiality, target population size and vulnerability, was determined as 'Low' for the EAS evaluation. This risk level influences the evaluation design and the level of effort allocated for the completion of the evaluation. The EAS evaluation was subject to a 'limited design' evaluation and relied on few targeted lines of evidence.

The scope of the evaluation includes the four activity streams funded under EAS, i.e. EAP, SOS, OCISM and PSEPR, and covers the period from April 2009 to the end of March 2014. It should be noted that fitness to work evaluations, which are a component of the Public Service Occupational Health Program, are distinct from EAS and will be part of the Public Service Occupational Health Program evaluation scheduled for 2018-2019. The evaluation is focused on the consortium of client-departments currently under agreement with Health Canada, excluding the DND and VAC.

To assess the progress made towards the achievement of the expected outcomes, an outcome-based evaluation approach was used for the conduct of the evaluation, identifying unintended consequences and lessons learned, as appropriate.

The evaluation focused on the five core issues (Appendix 3) outlined in the Treasury Board of Canada's Policy on Evaluation (2009). To guide the evaluation process, specific questions corresponding to each of the core issues were developed based on program considerations. The evaluation questions guided data collection efforts according to the following lines of evidence, which were used to triangulate findings, using a balance of quantitative and qualitative methods:

Literature Review

The literature review examined academic literature, as well as grey literature from reliable sources, to gather background information, to document the need for the program and to identify key drivers/determinants of work performance and personal resilience.

Given the limited design and the focus on the core public service, articles pertaining to particular aspects of the EAS such as addictions or to specific populations, were screened out.

Document review

Government reporting documents as well as EAS administrative documents, e.g., utilization reports, satisfaction and follow-up surveys, crisis center data and OCISM incidents reports, were analysed.

Key informant interviews

Twenty-four interviews with key staff members from EAS, affiliate providers, other related government services and non-client departments were conducted to gather in-depth information, including individual perspectives on EAS functioning, on how these services relate to other services and on the impact of these services on users.

Survey of EAS departmental liaison staff

The survey was sent to 110 departmental coordinators/managers from departments under agreement with Health Canada or with another provider of employee assistance services. Fifty two departments responded to all questions with an additional 36 submitting incomplete questionnaires for a response rate of almost 50%, which is considered a high response rate in the field.

3.2 Limitations and Mitigation Strategies

Challenges are inherent in any evaluation and limitations can impact the evaluation findings. As such, mitigation strategies are used to ensure that the data collected produce a credible evaluation report with evidence-based conclusions and recommendations.

Table 2 below outlines the expected limitations, their potential impact on the evaluation and the mitigation strategies that were employed in this evaluation to limit these impacts.

Table 2: Limitations and Mitigations Strategies
Limitations Potential Impact Mitigation Strategies
Lack of baseline data Inability to measure impact of program over time Triangulation of other lines of evidence to substantiate or provide further information on data obtained.
Limited quality and/or availability of outcome data Inability to provide evidence based findings Triangulation of evidence supplemented with perception data from various informants.
Consent not provided by EAS clients to access their files Inability to provide evidence based findings on EAS outcomes A few interviews with EAS affiliate providers were conducted to use their perceptions as proxies.
Limited information available from departmental EAP contacts Inability to determine the EAP best practices in a Canadian Federal Government setting, or why departments switched or did not switch to EAS Triangulation of other lines of evidence to substantiate or provide further explanation on information obtained.

4.0 Findings

4.1 Relevance: Issue #1 - Continued Need for the Employee Assistance Services

Key Findings:

The prevalence of mental health problems or illness suggests that there continues to be a need for interventions such as those delivered through EAS and to paying more attention to the psychological safety of the work environment, a factor known to enhance results of interventions such as EAP.

As noted earlier, the Mental Health Commission of Canada7reports that one in five Canadians will experience a mental health problem or illness in any given year with nearly a quarter of the country's working population being currently affected by mental health problems or illnesses leading to absenteeism, "presenteeism" and turnover. Approximately 30 per cent of short- and long-term disability claims in Canada are attributed to mental health problems and illnesses with adults in their early and prime working years being among the most vulnerable.

Mental health in the workplace is expressed as a feeling of balance, satisfaction and effectiveness in the work environment. Psychological health captures issues such as stress and anxiety, feeling overwhelmed either by work or personal issues or by the interconnection between the two, the lack of resiliency or coping skills or depression. When stress and anxiety are un-treated, they can develop into long term and more complex mental health issues and even long term disability.

According to the Mental Health Commission of Canada, many factors play a role in an individual's psychological make-up with both the workplace and the individual having a shared responsibility for maintaining and improving well-being because of the diversity of influences on a person's psychological health8.

Several individual characteristics are integral to employees' ability to cope with their various job demands such as self-esteem, self-confidence and higher optimism, a strong sense of self-efficacy, control and emotional stability. Increasing employees' leadership qualities and helping them set and attain broad goals will affect their work environment, increasing their job resources and resilience to job demands.

In 2010, Akhtar and Lee proposed an integration of two frameworks into the Job Demands and Resources model (JD-R), which can be applied to various work environments and assumes that job characteristics can be categorized as demands or resources.

According to this model, job demands tend to be aspects of the job that require sustained physical and/or psychological effort or skills and are therefore associated with certain physical and/or psychological costs9. Overall, job demands appear to be associated with negative health and negative organizational outcomes. For example:

  • workload increases seem to be associated with emotional exhaustion and a decrease in job satisfaction10;
  • the stress stemming from daily hassles appear to be a significant predictor of counterproductive behaviours and job dissatisfaction11;
  • role overload tends to be associated with burnout and turnover intentions12; and
  • negative environment (e.g., bullying) emphasizes other negative stressors which in turn, exacerbate negative health and organizational issues13 and can exacerbate effects of job demands on physical exhaustion, depression and uncertified absences14.

Interviews with affiliate providers for this evaluation confirmed that workload, emotional demands, rapid organizational changes, workplace environment, and changing expectations were all sources of stress reported by EAP clients15. The conservation of resources theory further states that "loss is more salient than gain", which suggests that employees are more sensitive to the negative effects of job demands than to the job resources they receive. Under the JD-R model, the exhaustion process briefly described above is to be balanced by the motivation process which builds on job resources to generate motivation, which leads to better health and organizational outcomes16,17.

Job resources are aspects of the job that are either functional in achieving work goals, reducing job demands, or stimulating personal growth, learning, and development18 that tend to be associated with positive health and organizational outcomes. Job resources include control over the task and autonomy, the absence of role conflict, positive workplace relationships including social support and management style19,20,21,22,23,24, 25, 26,27.

Unsurprisingly, the literature and expert advisors suggest that interventions that seek to increase employee resilience should not be exclusively focused on individuals' psychological well-being and preparedness but also intervene proactively on workplace attitudes. In fact, it seems that workplace group interventions that focus only on individuals' attitudes at work will have limited benefits, as psychological well-being and preparedness affect how an employee is able to handle various job demands28. Rather, interventions such as coaching and individual counselling, such as those provided through EAP (in addition to organizational services), are aimed to develop effective social coping strategies. Such strategies are known to increase self-esteem, itself associated with higher psychological well-being and higher optimism, resilience, and a stronger confidence, increasing psychological preparedness, which can be used as a personal coping strategy29.

According to the Mental Health Commission of Canada:

"The strategic pillars of a psychological health and safety system are prevention of harm (the psychological safety of employees), promotion of health (maintaining and promoting psychological health), and resolution of incidents or concerns. It has been well demonstrated that it is important to provide a psychologically safe work environment before health promotion endeavours can have significant success".

This suggests that while evidence gathered from the literature report that short-term counselling, such as EAP, is pertinent in helping managers and employees (and sometimes their families) enhance their resiliency skills at an individual level, the results of such intervention would be enhanced in presence of a psychologically safe work environment. The case of OCISM and PSEPR are telling examples in this respect. Influencing the psychological safety of work environment is an area where SOS could contribute by focusing the workshops on raising management awareness on the topic and on educating them on their role in protecting their staff psychological health.

While OCISM services can help nurses cope with the unique challenges associated with working in remote and isolated communities, the challenges remain. The particularity with nurses working in remote and isolated communities is that the threats to the psychological safety of the work environment often originate from their patients or from societal issues in the communities. The essence of nurses' functions is to help these patients also impacted by these societal issues. Addressing these issues at the community level is beyond the capacity of nurses and is the mandate of other FNIHB programs.

In the case of PSEPR, psychologically unsafe environment are likely inherent to circumstances triggering an emergency-type of responses, which are difficult to keep under control.

As it concerns the core federal public service though, expert interviews noted that mechanisms should be in place to assess the psychological safety of the work environment and that interventions, such as EAP should be considered as the minimum.

In fact, the Government of Canada committed to enhance the wellness and well-being of its employees through the Workplace Wellness and Productivity Strategy (2014) which will be presented in more detail in section 4.2 This Strategy emphasizes that workforce well-being generates higher levels of employee engagement, which in turn leads to better performing workplaces. The Strategy rests on a conceptual model (Figure 1 in section 4.2) that includes three phases. The first phase is about prevention and requires that risks causing absences from work be prevented, and is thus aligned with the concept of psychological safety promoted by the Mental Health Commission of Canada.

While interventions, such as EAP, contribute to the Strategy by helping employees so that they can remain focused and feel a lesser need to take time away from work to deal with personal or workplace issues, adjustments to increase SOS content, reach and delivery approach could also support the objectives of the Strategy, developed by Treasury Board. These changes could focus on educating managers on how to provide a psychologically safe environment, e.g., management/social support and workload management, an approach validated by interviews, where a representative from a corporation under contract with a major service provider noted that workshops aimed at fostering mental wellness in the workplace were integral to their strategy.

While managers have an important role to play in supporting their employees, it appears that they in turn need support as demonstrated in the 2012 annual Association of Public Service Executives report. The report notes that executives at all levels, who are also eligible to EAS, sought counselling from physicians and psychologists to deal with work-related stress issues30. These results suggest that improvement to the psychological safety of executives may also be necessary, particularly when considering that the proportion of executives using professional counselling services increased from 15.2% in 2002 to 20.6% 201231.

According to EAS utilization reports of the past five years, the main reasons for seeking EAP services were psychological health and family related issues (couple and children), which together accounted for over 75% of EAP users. Work-related issues accounted for approximately 15%. However it should be noted that definitions used for data tracking are not currently standardized.

In the five years covered by this evaluation (2009-10 to 2013-14), the utilization rate by employees of the EAP for client departments and agencies has ranged between 6.5% and 7.7% which indicates a continuing need for counselling services32. The number of suicide calls to the CRC also increased from 124 in 2010 to 238 in 2012, then started decreasing the following year with 209 in 2013 and with 45 in the first three months of 2014. Similarly, the EAP highest utilization rate was also observed in 2011-2012. While this further demonstrates the continuing need33 for the services, these results may also be influenced by the 2011 budget announcements which were validated by program sources.

The need for OCISM targeted to FNIHB nurses also remain. Between 2009-10 and 2012-13, the number of requests for OCISM services increased from 1,842 to 2,940, representing almost a 60% increase over four years. During the same period, the number of Critical Incident calls, i.e. calls related to an event or occurrence that triggered strong reactions and impacted on nurses' ability to provide care, to OCISM ranged from 30 to 50 calls34.

4.2 Relevance: Issue #2 - Alignment with Government Priorities

Key Finding:

EAS aligns with current government priorities to enhance the wellness and well-being of employees in order to increase productivity and reduce absenteeism. However, the current context seems to present opportunities to improve information provided to senior decision makers in further promoting mental health.

Based on Government of Canada documents, the Government of Canada aims to work at enhancing the wellness and well-being of its employees as referenced in the 2013 Speech from the Throne and Budget 2013. This commitment is demonstrated through TBS' ongoing efforts to modernize the disability and sick leave management system, with a view to ensuring that public servants receive appropriate services that support a timely return to work.

In 2014, TBS launched the 'Workplace Wellness and Productivity Strategy' as one of the tools proposed by the government, and to which EAP and SOS can contribute, to prevent issues impacting workplace wellness and productivity and to support early return from illness.

Figure 1: Disability Management: Prevention, Support for Recovery and Accommodation

Figure 1

The Workplace Wellness and Productivity Strategy specifically indicates that employee assistance programs, delivered by Health Canada or another service provider, should be enhanced with greater emphasis on prevention, education and awareness35. In the future and in parallel with other wellness measures, the EAP will be expected to contribute to the prevention of issues impacting workplace wellness and productivity. The additional services and prevention tools such as voluntary health risk assessments, online questionnaires and interactive information content currently being considered by SHSD are aligned with TBS directions set in the strategy and carry significant potential in informing senior decision makers on the health, either physical or psychological, of their employees. The development of such an information base would contribute to the evidence base necessary for senior decision makers to act more proactively, along with the enhanced role proposed for the SOS component in section 4.1.

The National Standards for Psychological Health and Safety in the Workplace (2013) describe the data that would be needed to appropriately track the issue within the workplace. At this point, these data are housed within various corporate services, e.g., human resources, values and ethics services, prevention and resolution of harassment, informal conflict management office, in most federal departments. Based on reviewed evidence, it is not clear whose role it is to aggregate and interpret this information from a mental health perspective and to bring emerging issues to the attention of departmental senior decision makers within each department. Considering federal and Health Canada roles relating to EAS, which will be discussed in section 4.3 below, this may provide EAS with an opportunity to broaden its role in promoting mental health.

4.3 Relevance: Issue #3 - Alignment with Federal Roles and Responsibilities

Key Finding:

As an employer, the federal government and Health Canada have clear legislated roles relating to the health, safety and well-being of federal employees.

While other services provided across the government and within departments generally complement services offered by EAS, there may be overlap (similar services) of services for specific subsets of the target population, e.g., executives.

4.3.1 What are the federal and Health Canada roles relating to EAS?

The federal government has clear roles and responsibilities with respect to promoting and preserving the health (including the mental health) of its employees, as established in the following documents:

  • The 1994 Treasury Board Policy on Occupational Safety and Health, which states that departments must provide employee assistance services to its employees.  This includes both psychological and physical health36.
  • The 1999 Treasury Board Policy on Employee Assistance Program which requires that Departments provide employees with EAP services that conform to the policy37.
  • Various collective agreements. For example, article 22 - Health and Safety of the current collective agreement with Program and Administrative Services workers indicates that the Employer shall make reasonable provisions for the occupational safety and health of employees, including both physical and mental health38.
  • The Department of Health Act, subsection 4(2)(f) provides that the Minister's powers, duties and functions relating to health include "the promotion and preservation of the health of the public servants and other employees of the Government of Canada"39.

EAS offers services to the entire federal public service and aligns with TBS Sub-program 1.2.3: Comprehensive Management of Compensation and the Policy Framework for the Management of Compensation40,41. TBS 2014-2015 Report on Plans and Priorities' planning highlights note that "efforts will be pursued to advance the development of a modern approach to public sector pensions and benefits [...], including modernization of the disability and sick leave management system, to better support employee productivity, recovery and wellness", as discussed as part of Government priorities' section.

The 1999 Treasury Board Policy on Employee Assistance Program also requires departments to establish procedures to: "provide an initial stress debriefing session as soon as possible after such an incident [critical incidents related to the nature of their work] and EAP follow-up if required". For First Nations and Inuit Health Branch nurses, OCISM is the procedure established by the department.

The Psychosocial Emergency Preparedness and Response Team is a similar approach aimed at providing support to federal personnel exposed to traumatic stress resulting from a disaster or major event. Both the Office of Emergency Preparedness and the PSEPR Services hold key roles in fulfilling Health Canada's emergency management responsibilities42.

4.3.2 Are there other programs that complement, overlap or duplicate the objectives of EAS?

The Public Service Health Care Plan (PSHCP) offered to federal employees complements EAP by providing coverage for employees to seek private providers for short or long term help related to mental health issues and psychotherapy.  The PSHCP has a cap for psychotherapy services.  Employees pay a portion of the monthly PSHCP premium and generally pay 20% of the costs for psychotherapy up to the cap and then pay 100% above the cap.  As well, the employee needs a doctor's prescription for the PSHCP psychotherapy to be covered.

As noted earlier, the TBS Workplace Wellness and Productivity Strategy places EAS within a large suite of services available to promote wellness and productivity within the federal Government of Canada. The Strategy views EAS as one prevention component of a larger comprehensive wellness strategy.

The Advisory Service for Executives (ASE) is a non EAS/SOS program to help executives.  The ASE provides confidential service available free of charge to all federal executives nationally and abroad. The number of individuals who used the service rose from 232 in 2011-2012 to 306 in 2012-201343, an increase of 32%. Considering there were 6923 executives in the federal public service in 2012-2013, the utilization rate for this year was 4.4 per cent44 for the executive population. During this year, 260 executives used EAP services as well which demonstrates that executives uses both services.

The ASE is supported financially by the Deputy Head community and housed at Association of Public Service Executives. ASE was established in 2003 in response to expressed needs of executives for an objective and confidential ear in dealing with difficult work-related situations. It is a single window for arm's length advice and referral to a specialized resource network. It is intended to complement other services for public service executives, e.g., Public Service Commission Executive Counselling Services. ASE's services include supporting executives who feel stress from sources such as increased work demands, reduced resources and unhealthy work environments.

4.4 Performance: Issue #4 - Achievement of Expected Outcomes (Effectiveness)

The section below is aimed at reporting on EAS outcomes. Challenges were encountered mainly associated with the availability of data. For example, while the number of employees is known, the number of dependents is not known which means that the overall population cannot be determined. For other components (SOS, PSEPR and OCISM), limited relevant data were available to document their achieved outcomes and collecting such data would have required more resources (in time and money) than what was allocated for this low risk evaluation.

Most EAS activities rely on some form of counselling to build on the strengths of individuals to enhance their resiliency skills while in the case of SOS, changes anticipated are more aligned with stages involved in knowledge translation. Please refer to section 2.3 for a detailed description of the logic model (Appendix 2).

4.4.1 Access to Knowledge, Tools and Services

Key Findings:

The number of calls increased over the evaluation period, although the growth is not consistently aligned to the growth of the consortium population. The proportion of family members accessing EAP services as well as the number of requests for SOS sessions (support to managers and teams) and OCISM services also appear to be on the rise.

EAP

Table 3 below provides information on the evolution of the EAP population size, CRC activity levels and EAP utilization.

Table 3: EAP utilization data
Year 2009-10 2010-11 2011-12 2012-13 2013-14
Consortium Population 113,934 131,142 135,678 166,361 171,655
Calls to CRC 35,586 38,122 42,373 46,289 48,401
Suicide-related call 99 114 204 238 194
Immediate counselling (not suicidal) N/A 1,055 1,056 1,107 760
Referrals to network 13,686 14,847 16,257 16,576 15,318
Employees using EAP (Utilization Rate) 8,351 (7.3%) 9,657 (7.4%) 10,489 (7.7%) 11,707 (7.0%) 11,143 (6.5%)
Family Members Using EAP (Proportion of total users) 1,079 (11.4%) 1,262 (11.6%) 1,222 (10.4%) 1,429 (10.9%) 1,806 (13.9%)
Total Utilization Rate 8.3% 8.4% 8.7% 8.0% 7.6%
Source: Employee Assistance Services

According to Table 3 above, the consortium population has increased by 51% since 2009-2010. In 2012-13, EAS covered approximately 70% of the core federal public service employees and their dependents. As noted earlier, the consortium employee population refers to the core public administration and separate agencies, excluding billed departments given the weight they represent and the extent to which their service utilization patterns differ from the general consortium.

The number of calls to the CRC is used here as proxy for EAP awareness. While the number of calls increased from 35,586 in 2009-2010 to 48,401 in 2013-14, the annual growth in the number of CRC calls is not aligned with the growth of the consortium population. In 2010-2011 for example, the consortium population increased by 15% while the calls made to the CRC only increased by 7%. The same would apply for 2012-2013. This lag may be due to the timing a department joining the consortium or to a lack of awareness associated with a change in service provider.

The situation was reversed in 2011-2012 when the consortium population and the number of calls made to the CRC increased by 3.5% and 11%, respectively. As reported by interviewees and program documents, this data likely illustrate the impact of the 2011 Budget announcement on EAP utilization. Data on the number of employees aware of EAP or any other EAS component are not currently available to provide a more precise indication of awareness of available services.

According to EAP data, the proportion of EAP users out of the number of referrals to the network is also on the rise, increasing from 61% in 2009-2010 to 72.7% in 2013-2014. This may suggest that there are fewer stigmas associated with seeking the type of help provided by EAP.

While the discussion above provides some indication of service awareness, not all individuals initiating a call to CRC end up actually using EAP counselling services. Service utilization is defined in various ways in the literature. For the purpose of this evaluation, it consists of the number of users (federal employees and their dependents) divided by the population of federal employees covered by EAP. Most of the time, we will refer to 'employee' utilization given that the EAP long-term outcome is expected to trigger a change mainly in this group. In addition, it is not possible to identify a reliable figure for the population, when eligible dependents are included.

The utilization rate by employees of the EAP client organizations (excluding public servants' family members, DND and VAC) rose from 7.3% in 2009-2010 to 7.7% in 2011-2012 and declined to 6.5% in 2013-2014. There is no apparent trend when data are considered longitudinally and in association with the growth in the consortium population. While in 2010-2011, the increase in utilization reflected the growth of the consortium population, the increase in utilization reported for 2012-2013 is half the growth of the consortium population.

The proportion of EAS users being employees' dependents ranged between 11.4% from 2009 to 13.9% in 2013 (excluding DND), after a small decline in 2011-2012 and 2012-2013. No information is currently available to measure awareness of the availability of EAP among the consortium population and its dependents.

SOS

As noted earlier, SOS provides customized training sessions and workshops to employees as well as long-term coaching to managers and executives. The workshops and coaching offered by SOS are provided on a fee for service basis and thus requires management pre-approval.

In the absence of data on managers' awareness of services available through SOS, the number of requests by managers is in this case used as a proxy, reporting on access to SOS interventions or sessions.

Table 4: SOS Sessions
Locations 2009-10 2010-11 2011-12 2012-13 2013-14 5-Year Annual Average
NCR 189 240 291 302 263 257
Non NCR 138 181 253 171 172 183
Total 327 421 544 473 435 440
Sources: SOS Mini Tracking documents

The SOS Mini tracking documents show that SOS requests rose by 66% from 2009-2010 to 2011-2012. Increased activity over this period is likely associated with the stress stemming from the cuts in the federal public services that were announced in the federal budget released in 2011. In some way, this increase in SOS requests may have contributed to raise managers' awareness about this component which in turn may explain, at least partially, why the number of service requests remained high after 2011-2012, although declining. Other possibilities could be that post-budget organizational issues are still being worked out or that the increase in the number of requests is an illustration of the increased client base.

OCISM

According to the OCISM Annual Summary Report (2013), requests for services which are defined as the occurrence of an event reported to the OCISM response centre consistently increased from 2009-2010 to 2012-2013 with 1,842 and 2,940 calls received, respectively. This 60% increase over such a short period of time suggests that nurse awareness of and/or needs for OCISM services increased.

PSEPR

As noted earlier, PSEPR is the smallest component of EAS, which receives only $130K of A-Base funding annually. PSEPR refers to mental health support for federal employees in the occurrence of an event requiring an emergency type of response, including natural disasters, major accidents and terrorist attack. Government employees may need psychological or emotional support when experiencing the consequences of such events. While there has not been any event over the evaluation period, the team participated 'on standby' to a number of events including the Vancouver Olympics (2010) and the G8/G20 Summit in Huntsville (2012).

4.4.2 Enhanced Awareness and Use of Resiliency Skills

Key Finding:

Affiliate providers' observations and users' self-reported assessments indicate that EAP and OCISM contribute to improvement in self-awareness, client resiliency and coping skill.

Insufficient data is available to report on SOS and PSEPR impacts.

EAP

Although extensive data on EAP utilization is available, there is currently no mechanism in place to measure the awareness and knowledge of resiliency skills amongst the target population prior to their exposure to EAP services.

In this context, it is assumed that employees' awareness and knowledge of resiliency skills is being enhanced through their utilization of EAP counselling services, the rationale being that when employees access the services, they gain knowledge and enhance their awareness of resiliency skills (as self-reported in user surveys).

In 2014, the awareness and use of resiliency skills subsequent to the provision of EAP services was assessed for the first time through a set of a few questions added to the annual telephone survey conducted by SHSD. According to this survey, a majority of respondentsi saw changes, at least moderately, in their self-coping abilities since the end of EAP sessions:

Table 5: Telephone Survey Results March 2014 Self-Coping Abilities
Question: Since the end of your EAP sessions, has there been any change with: None at all Slightly Moderately Quite a bit Extremely N/A
1. Your ability to deal with stressful circumstances? 24.5% 23.5% 21.0% 25.5% 4.5% 1.0%
2. Your capacity to put things into perspective? 21.5% 20.0% 23.0% 29.5% 4.5% 1.5%
3. Your sense of capability (i.e. your capacity to influence the course of events)? 26.0% 18.0% 23.5% 25.5% 5.5% 1.5%
4. Your ability to act positively when addressing difficulties? 23.0% 17.5% 24.0% 29.0% 6.0% 0.5%
5. Your use of these above skills at work? 30.5% 13.0% 23.0% 19.5% 4.5% 9.5%
Source: 2013-2014 Telephone survey results, EAS, Health Canada

Interviews conducted with EAP affiliate providers (n=12) support survey results. Interviewees observed that the clients seen in individual sessions were becoming more aware of their strengths and resiliency skills and were able to make some attitude and behaviour change, potentially because of their exposure to EAP45. For example, affiliate providers reported that clients were able to improve their motivation, decrease their anxiety, achieve some state of inner peace as well as learn where to find resources related to communication or problem solving.

SOS, OCISM and PSEPR

Increased use of knowledge and tools gained from SOS sessions was not assessed given there is no systematic follow-up conducted to measure the uptake of the knowledge and tools obtained through workshops and sessions delivered through SOS.

While evidence date back from 2010, an evaluation of an OCISM pilot project suggest that OCISM significantly affected nurses coping skills. However, the evaluation found insufficient information to effectively measure PSEPR service outcomes.

4.4.3 Influence on Work Satisfaction and Productivity

Key Finding:

Self-assessment data suggests that EAP and OCISM contribute to an improvement of employees' mental and physical health as well as productivity.

Since 2010-2011, the EAP Telephone Surveys ask respondents how their productivity had been affected by their emotional health during the four weeks preceding their exposure to EAP, compared to after the end of the sessions. Results are presented in Table 6 below46

Table 6: Change in Reported Productivity level (as Affected by Emotional Problems) before and after exposure to EAP
Question: During the four previous weeks before you contacted EAP, to what extent had you accomplished less than you would like in your work or other daily activities as a result of emotional problems (such as feeling depressed or anxious)? And after your sessions?
Results 2010-2011 2011-2012 2012-2013 2013-2014
Not at all +99.2% +135% +109% +135%
Slightly +89.8% +73.7% +86.1% +54.0%
Moderately -12.9% -39.3% -31.6% -17.2%
Quite a bit -58.4% -76.0% -64.4% -62.2%
Extremely -76.4% -81.8% -75.0% -76.3%
Source: Telephone survey results, EAS, Health Canada

Results of these surveys consistently report significant improvements in self-reported overall productivity and reduced absenteeism. In 2013-14, 40.7% of clients (n=410) reported having experienced "quite a bit" or "extreme" difficulty in performing their work or other regular daily activities as a result of their emotional problems in the four weeks before they contacted the EAP. This proportion dropped to 13.2% after the end of EAP sessions, which represents a 68% reduction in the number of survey respondents with low productivity due to emotional issues. As well, there are also consistent increases in the proportion of respondents who reported having experienced less difficulty in performing their work, or other regular daily activities, due to emotional problems after their involvement with EAP.

This data suggests that survey respondents have used the resiliency skills gained during their involvement with EAP to increase productivity.

Similar results were reported on productivity as affected by physical health for the period covered by the evaluation. In the same year and according to the same survey, 36.6% of respondents reported considering their overall health to be "fair" or "poor" at case opening. This proportion dropped to 16.5% at case closing, which represents a 55% decrease in survey respondents reporting a low health status. At the other extreme of the continuum, 31.7% of respondents reported their overall health (Table 7 below) to be excellent or very good before using EAP, the proportion of which increased to 59.3% once their sessions ended. There was a significant increase (87%) in number of respondents reporting an improvement in their perceived health status following the use of EAP.

Table 7: Change in Perceived Health Status before and after exposure to EAP
Question: During the four weeks before you contacted EAP, how would you say your overall health was on a scale of 1-5, where 1 is excellent and 5 is poor? And after all of your sessions ended?
Results 2010-2011 2011-2012 2012-2013 2013-2014
Excellent +75.8% +117% +90.9% +109.5%
Very Good +77.1% +78.1% +95.3% +75.9%
Good -12.1% -23.5% -9.5% -25.2%
Fair -53.3% -48.1% -53.8% -40.1%
Poor -65.7% -71.7% -67.2% -76.6%
Source: Telephone survey results, EAS, Health Canada

Table 8 below displays Telephone Survey results pertaining to self-reported absenteeism. In 2013-14, 45% of respondents reported missing 1 to 30 days of work before EAP (30 days prior to contacting the service), and 23% reported missing 1 to 30 days after (45-60 days since beginning EAP). Clients reported at case opening that they had been absent on average 2.85 days in the 30 days prior to using the EAP. At case closing, clients reported being absent on average 1.61 days at the end of their EAP sessions - which represents a 43% drop. Although there are no readily available cost estimates of the benefits of EAP for productivity, Attridge estimated that every dollar spent on EAP programs produces an increase in productivity of $647.

Table 8: Change in Self-reported Absenteeism before and after exposure to EAP
Question: In the 30 days prior to contacting your EAP, how many days have you been knowingly or unexpectedly absent from your job because of your problems (average number of days)? And after all of your sessions ended?
Results 2010-2011 2011-2012 2012-2013 2013-2014
0 days +32.5% +32.0% +42.1% +36.6%
1-3 days -40.4% -52.9% -33.3% -35.4%
4-7 days -26.6% -58.2% -84.2% -78.3%
8-12 days -82.4% -66.7% -80.0% -50.0%
13-20 days -22.2% -100% -92.1% -85.7%
21-30 days -30.0% -7.1% +34.9% 0%
Source: Telephone survey results, EAS, Health Canad

No changes were observed in 2013-2014 in the portion of users reporting missing 21 to 30 days before and after EAP sessions. These can include employees who might be on stress or disability leave or who already have a pre-existing mental health illness that requires treatment and long term therapy beyond EAP's short term mandate. Research shows that people with mental disorders such as anxiety and depression are more likely to experience prolonged absence from work and take multiple sick days.

The reduction in time away from work is significant. Looking at the 2013-14 survey results, an average of 1.24 days "saved" in a month corresponds to a total of 6100 avoided missed days from work over a year by the 410 EAP users. That corresponds to 23.5 full-time equivalent employees. These results indicate considerable potential savings to the employer and enhanced workplace performance.

SOS, OCISM and PSEPR

The contribution of SOS, OCISM and PSEPR to work satisfaction and productivity was not assessed given there is no systematic follow-up conducted to measure the uptake of the knowledge and tools learned through workshops and sessions delivered through SOS.

While evidence date back from 2010, an evaluation of an OCISM pilot project suggest that nurses self-reported work performance, stress levels and sense of well-being improved as a result of OCISM. However, the evaluation found insufficient information to effectively measure PSEPR services outcomes.

4.5 Performance: Issue #5 - Demonstration of Economy and Efficiency

4.5.1 Are EAS activities delivered in an efficient manner?

Key Finding:

Services meet industry-recognized quality standards. There is a high satisfaction rate from users, affiliate providers and client-departments with the timeliness and quality of services.

While concerns were raised by affiliate providers about administrative procedures being burdensome (e.g., time for payments to be processed and burden of administrative processes) due in part to obsolete reporting and billing systems, these controls may be necessary for an economic use of funding.

EAS is accredited by the COA. To obtain this accreditation, EAS had to demonstrate and maintain the processes and service standards required by the accrediting organization. EAS reached level 5 (full implementation) of the requirement in March 2012 and received, in September 2013, its third consecutive accreditation via the COA. Accreditation from COA signifies the highest level of service quality and organizational performance in the EAP industry. The focus of the accreditation process is on management and delivery. Confidentiality and privacy, legal compliance, safety and security, financial management practices and quality of service delivery are examples of the various aspects examined.

The accreditation process provides a first indication that most infrastructures and procedures are in place for sound management and efficient service delivery. This is supported by the satisfaction levels reported in the telephone surveys of EAP clients that have been conducted in the past 5 years. Survey results show a high level of satisfaction with how services were delivered. In 2013-2014 (n=410):

  • 96.5% of respondents were very satisfied or satisfied with the counsellor who initially responded to their telephone call
  • 87.5% were very satisfied or satisfied with the EAP counsellor's ability to understand their concerns
  • 81% were very satisfied or satisfied with the help they received in dealing with their concerns
  • 88.5% were very satisfied or satisfied with the quality of services they received
  • 96% would recommend EAP to a co-worker in need of similar help.

The views of affiliate providers interviewed (n=12) are consistent with those of respondents to the EAP telephone follow-up survey and EAP departmental liaisons surveyed (n=65). Key EAP strengths identified through interviews included the quick turnaround to arrange sessions, 24/7 accessibility, consistent excellence of service, the responsiveness to clients' needs and the recruitment and selection process of affiliate providers.

In terms of data tracking, EAS produces utilization reports which are provided to each client-department to give them basic information on EAS service use within their organization. While utilization reports are considered as a valuable source of information (survey administered to clients and non-clients departments), these reports are mostly used for accountability of resources and to have an overall appreciation of possible trends in users' categories. Interviewees noted that EAS information was not in itself sufficient to draw conclusions on employees' well-being and should be supplemented with data on turnover and absenteeism, for example.

There is an expectation that EAS data should separate results for employees versus family members, and take a stronger role in evaluating and identifying potential risk areas in an organization (e.g. a specific branch or category of employees). For this to happen, EAS would need to rely on more modern data systems given the current one does not allow for the completion of timely in-depth analysis. There are challenges associated with balancing the need for cross-reference analysis while ensuring respect of confidentiality and privacy. Minor adjustments could be made to current data collection tools and analysis methodologies to produce reports that would better inform and support senior management decisions in providing services responsive to their employees' documented needs.

Administrative concerns were expressed by affiliate providers. Those may be areas for potential improvement:

  • The time elapsed from seeing a client to the receipt of payment, could be up to two months and would be appreciated if shortened;
  • The process of preparing and submitting the plan for going beyond three sessions currently places an administrative burden on affiliate providers and could be streamlined;
  • Affiliates are not compensated for follow up telephone calls to clients or even for telephone calls to Health Canada related to cases; and
  • According to affiliate providers, the number of remaining sessions is not always communicated to affiliate providers leading to billing confusion.

While some of the issues identified above may originate in the lack of modern online billing and reporting system reported by many affiliate providers, these issues also refer to mechanisms necessary to have some control over the delivery of service and to ensure an economic use of public funds.

4.5.2 Are EAS activities delivered economically?

Key finding:

The use of a consortium pricing model allows client-departments to identify costs from year-to-year for the provision of the employee assistance program. While there may be some indication of economy of scale, data available are not sufficient to formally conclude as to whether such economies exist.

The Consortium Pricing Model

In accordance with the Consortium Pricing Model, Health Canada enters into agreement with federal departments and organizations interested in services delivered by Health Canada. Although there may be departmental particularities, these agreements define the eligible population and the services to be provided under the agreement and at what cost. For EAP, a fixed fee per employee is charged to the organization up to an eight per cent utilization rate. The extra fees that apply when the eight per cent threshold is exceeded are also described in the letter of agreement. The few departments which are billed according to their use of EAP are exceptions to this arrangement. According to SHSD, few small departments may be charged a higher fee per employee.

The use of a Consortium pricing model is intended to allow all federal departments to benefit from economies of scale, while being able to predict costs from year-to-year for the provision of employee assistance programs"48.  The Consortium Model provides member organizations a form of "insurance" by capping the amount of surcharges that could be invoiced to EAS customers if their EAP annual utilization rate were higher than the negotiated Consortium protected rate. This option is appreciated by client-departments and agencies because it provides organizations with added financial security and increased planning capability49.

The analysis of economy, or of the extent to which the intervention is delivered economically, requires comparative data which are not currently available likely due to the reluctance of vendors who consider this information as proprietary (Attridge, 2013). In his study, Attridge further notes that the field lacks a set of standard definitions to use in determining relevant benchmarks in service delivery and outcomes. While the profile of external EAP vendors published in 2013 attempted to fill some of these gaps, the study still does not provide the information that would be necessary for a meaningful assessment of the economy of the program delivered by Health Canada. For example, while the program delivered by Health Canada seems to be within the industry average in terms of the number of counselling sessions per case, there is great variance in the industry as it pertains to this variable; the same applies for utilization.

In this context, we tried to consider how the EAP delivered by Health Canada evolved over time in terms of eligible population, revenues, salaries and staff and to consider this information in relation to others. However, it should be understood that this analysis only provide a rough assessment given the difficulty associated with disentangling the contributions to the different components of EAS or the manner in which the consortium model is being implemented, i.e. departments being temporarily or permanently out of the consortium based on their usage patterns.

The EAP client base has steadily increased since 1992 which should theoretically reduce the cost per employee assuming a constant utilization rate. Looking at the last five years, the Consortium employee population increased from 113,934 in 2009-10 to 171,655 in 2013-14. This represents a 51% increase in the consortium population while the revenues increased by 47% and the number of users increased by 38% (including family members).

An overview of EAP revenues for the period comprised between 2009-2010 and 2013-2014 is provided in Table 9 below.

Table 9: Overview of EAP Revenues
Categories 2009-10 2010-11 2011-12 2012-13 2013-14
Total EAP revenues $4,834,512 $5,742,304 $5,807,944 $6,416,360 $6,408,911
EAP RevenuesFootnote* $4,834,512 $5,270,304 $5,520,816 $6,416,360 $6,408,911
Revenues from over utilization $0 $472,000 $287,128 $0 $0
Consortium Employee Population 113,934 131,142 135,678 166,361 171,655
Employees using EAP (Utilization Rate) 8,351 (7.3%) 9,657 (7.4%) 10,489 (7.7%) 11,707 (7.0%) 11,143 (6.5%)
Family Members Using EAP 1,079 1,262 1,222 1,429 1,806
Average cost per user (including family members) $512.67 $526.38 $495.93 $488.46 $549
EAP fee per covered employee (excluding revenue from utilization) $42.43 $40.19 $40.69 $38.56 $37.33
EAP fee per covered employee considering overutilization $42.43 $43.78 $42.81 $38.56 $37.33
Source: Employee Assistance Services

Return to footnote* referrer EAP revenues exclude revenues from billed departments, although the RCMP was integrated to the consortium in 2013-2014.

EAS salaries and operations are funded through cost-recovery transactions (except for PSERP which is A-Base funded). Revenues are based on $42 per federal employee covered under interdepartmental letters of agreement although very small departments may be charged a higher amount. According to figures included in Table 9, the fee per employee covered decreased slowly as the consortium population increased. Considering a supplemental cost of $4.00 per employee for each percentage point or part thereof when the threshold of 8% utilization is exceeded, EAS does not seem to assume a financial risk should utilization increase significantly. Although it was beyond the scope of the evaluation to do a detailed comparative analysis with other similar plans, interviews with non-client organizations suggested that other plans sometimes included the provision of workshops in their fee per employee. Adding SOS revenues, for which the annual average is $1.86M, would increase the fee charged per employee. In this context, a detailed comparative analysis would be necessary to conclude as to whether EAS actually provide services at the lowest possible cost to the federal public serviceFootnote50. As suggested by Attridge, such detailed analysis would present significant challenges and would require a study on its own, which was beyond the scope of a low risk evaluation.

The Health Canada EAP was nonetheless compared to the Region of York for which a report was released in 2007. According to this report, which documents renewal of EAP services for its employees (approximately 4,000), two short-listed suppliers offered EAP services for $39 and $41, respectively, per employee per year.  Although this information is somewhat dated and does not allow for the comparison of scope of services providedFootnote51, it would suggest that the EAP fee per employee is in line with this comparator.

Results from the survey to EAP liaisons in organizations (clients and non-clients) show that 50% of client respondents (n=46 under contract with HC completed questionnaire) indicated cost as being a reason why their organization chose HC as a provider for EAS, 70% chose HC for the service offerings, and 37% indicated that there has never been a compelling rationale to change provider.

Measures are in place to provide quality services to clients while minimizing the potential for abuse. Affiliate providers must provide a counselling plan to the national office by the end of the third hour of counselling to demonstrate that the case can be dealt with within the remaining five hours of counselling. Approval of the national office must also be sought for exceptional circumstances where more than eight hours of counselling are to be provided. Affiliates are compensated only for the billable services with no associated charges. Compensation for 'no-show' is limited to one by service user.

EAS Human Resources

A 2008 capacity management report indicated that sufficient capacity (e.g., infrastructure, staff resources and methods) exist within EAS to handle significant increases in short-term workload and moderate levels of longer-term workloadFootnote52. The population covered by EAP and information on EAS employees administering the program are displayed in Table 10.

Table 10: EAS Human Resources
Categories 2009-10 2010-11 2011-12 2012-13 2013-14
Consortium Employee Population 113,934 131,142 135,678 166,361 171,655
EAS Employees (FTEs)Footnoteii 39 37 49 54 58
Salaries and wages $2,910,126 $2,968,304 $3,104,327 $3,381,884 $3,544,987
O&M $5,042,905 $5,301,051 $5,665,159 $6,721,054 $6,597,175
Total $7,953,031 $8,269,354 $8,769,486 $10,100,786 $10,163,879
Source: Employee Assistance Services

Based on program records, the consortium population and the number of FTEs both increased by 51% and 49%, respectively. However, the amount spent on salaries only increased by 22% during this period which suggest that activity growth does not translate by an equivalent growth in expenditures related to human resources. As noted earlier however, this analysis can only provide a rough assessment of the situation given the difficulty associated with disentangling the contributions to the different components of EAS.

4.5.3 Is there appropriate performance measurement in place?

Key Finding:

As evidenced by the recent renewal of the EAP accreditation, the performance measurement system in place ensures the collection of administrative and operational information. The efforts underway to upgrade this system should integrate the collection of information necessary to track outcomes associated with the various EAS components.

Interviews with both affiliate providers and EAS management indicated that the information technology infrastructure does not provide EAS with a fully sufficient data management capacity and performance measurement system.  However, a large scale IT project to address this issue is currently underway.

In March 2012, EAS had reached level 5 [Full Implementation] of the requirement as the required measurable performance indicators were "operating as intended" and "implement fully in all intended areas of the organization"Footnote53. As noted earlier, the COA certification demonstrates that EAS maintains appropriate management practices and service delivery standards as it pertains to topics such as confidentiality and privacy, legal compliance, safety and security and financial management practices. While EAP operational data already collected is a very good start, current program delivery lends itself well to the collection and analysis of additional data that would allow for a more precise examination of EAP outcomes.

For example, a retrospective study of records maintained by affiliate providers can be contracted to an academic in the field in order to develop a more precise profile of service users and to document the results of the services rendered. This would necessitate the required consent and ethical review before proceeding but would nonetheless be possible.

Definitions currently used should also be reviewed to allow for a more precise coding of some data and ensure consistency.  For example, reason for consulting includes very broad categories such as 'psychological health' which is also collected slightly differently in a different tool.

Information currently collected on SOS seems to be those necessary to support billing and make the demonstration to client-departments as to how their funds were spent. More precise information on the type of workshop, the need triggering the request and the number of participants would be helpful. Surveys could also be administered at various points in time, e.g., before the workshop, immediately after and a few months after, to document workshop outcomes, their persistence over time and the impact in the workplace. Should such an undertaking be excessively demanding on program resources, a sampling plan could be considered.

Data currently collected for OCISM are limited. Considering the relatively small population size, additional and more regular data collection would allow for a better tracking of the situation as well as OCISM contribution over time. The information collected would probably be similar to EAP.

5.0 Conclusions

5.1 Relevance Conclusions

Based on the information reviewed over the course of the EAS evaluation, it can be concluded that there is a demonstrated need for interventions such those delivered by EAS and that it is the role of the Government of Canada, and especially of Health Canada, to protect the health and safety of federal employees.

All lines of evidence confirmed that mental health issues, such as anxiety, stress and depression, regardless of where or how they arise, can easily cross back and forth between the workplace and family life and impact on the wellbeing and productivity of individuals.

Considering that the federal government, and specifically Health Canada, has also clear roles and responsibilities with respect to promoting and preserving the health (including the mental health) of its employees, as established in the: Department of Health Act; Treasury Board Policy on Occupational Safety and Health; Treasury Board Policy on Employee Assistance Program; as well as in various collective agreements; it is evident that there is an ongoing need for some form of intervention to mitigate the impact of stress associated with life or work demands.

The Workplace Wellness and Productivity Strategy developed by the Treasury Board is a government priority aimed at ensuring federal employees receive the services necessary to minimize absenteeism and long term disability. The Strategy specifically indicates Health Canada's EAP as a component in the suite of services to deal with mental health issues and further note that EAP should have an enhanced focus on prevention which could be a potential expansion for SOS.

While there are a number of programs complementing EAS, most of these programs seem to have their own niche and do not duplicate EAS, although there seems to be some overlap with the Advisory Service for Executives.

5.2 Performance Conclusions

5.2.1 Achievement of Expected Outcomes

EAP and OCISM seem to contribute to the increased use of resiliency skills and tools and thus contribute to increased productivity for the users remaining at work and to a modest reduction in absenteeism. Data is not sufficient to conclude on the extent to which SOS' and PSEPR contributed to attitude or behaviour change.

There is evidence indicating that EAS increased its client base significantly over the course of the evaluation period. The increases in the proportion of calls to CRC were significantly lower than the increase in the consortium employee population. While no direct measure of awareness available, this data suggest a possible awareness gap in departments having recently joined the EAS consortium or that the department joined the consortium late in the fiscal year.

Based on data available from the program and collected through the interviews, EAP seems to contribute to the increased use of resiliency skills and tools to help federal employees in finding satisfaction at work. Data available also suggest that EAP contributes to increased users' productivity for those remaining at work and contributes to a reduction in absenteeism.

With respect to other EAS service lines, there is not sufficient data to conclude as to whether those services contribute to attitude or behaviour change after the intervention.

5.2.2 Demonstration of Economy and Efficiency

EAS is delivered in an economic and efficient manner.

For the most part, EAS does demonstrate efficiency and economy. EAS does deliver high quality services in accordance with industry and professional standards and has mechanisms in place to ensure economical use of public funds although updated technological infrastructures could contribute to greater efficiencies, e.g., for the collection of information and compensation of affiliate providers. When compared to one other jurisdiction, it appears that EAP fee per employee is in line with this comparator.

6.0 Recommendations

The evaluation identified the following recommendations:

Recommendation 1

EAS examine its role within the Treasury Board Workplace Wellness and Productivity Strategy to identify measures contributing to the prevention of issues potentially impacting mental health of employees.

Given the broad thrust of the Strategy, EAP may need to develop additional services and/or tools and enhance existing ones to further support the prevention, education and awareness objectives of the Strategy, such as health risk assessment and / or further promoting e-counselling. EAS should explore how SOS can more significantly contribute to support the objectives of the Strategy.

Recommendation 2

EAS examine what data should be collected and analyzed to develop a more fulsome analysis of the needs and impact of EAS on productivity of employees.

These analysis should be used by EAS to adapt its service offer and to inform senior management of client departments. This could include:

  • reviewing the management information and performance measurement systems to ensure data support the objectives of the Workplace Wellness and Productivity Strategy;
  • considering the standards developed by the Mental Health Commission of Canada as a possible starting point for EAS to refine its datasets; and
  • developing partnerships in client-departments that would allow EAS to communicate the results of their analysis to senior departmental representatives to inform their decision making.

Appendix 1 - References

2008-09 Health Canada Departmental Performance Report

2009-2010 EAS Consortium Report

2009-2010 OCISM Annual Report

2010-11  EAS Consortium Report

2010-11  OCISM Annual Report

2011-12 EAS Consortium Report

2011-12 OCISM Annual Report

2012-13 EAS Consortium Report

2012-13 Health Canada Departmental Performance Report

2012-13 OCISM Annual Report

2013-2014 EAS Consortium Report

2013-14 HC Report on Planning and Priorities

2013-2014 OCISM Annual Report

Agreement between the Treasury Board and the Public Service Alliance of Canada:  Program and Administrative Services (all employees) Expiry Date: June 20, 2014

Akkermans, J.,  Brenninkmeijer, V.,  Blonk R., and Koppes L. J.,  Fresh and healthy?: Well-being, health and performance of young employees with intermediate education. 2009

All calls - suicide related - crisis - monthly comparisons 2009 up- Dec 31 2013

APEX 2012 Executive Work & Health survey results

Attridge, M., et al.. EAP effectiveness and ROI EASNA Research Notes, Vol. 1, No. 3 .2009

Attridge, M.,  Cahill, T.,  Granberry S.W.,  Herlihy, P.A. The National Behavioral Consortium Benchmarking Study: Industry Profile of 82 External EAP Providers, 2013.

Bordeleau, P., 2012-13 Annual Report Advisory Service for Executives.

Capacity Assessment Workplace Health and Public Safety Programme Service Profile for Employee Assistance Services, 2007.

Capacity Management Action Plan_EAS_Appendix A (FEB 08)

Canadian Government Executive September 2013: Apex Survey 2013 - The Impact Of Work On Executive Health

CISM Annual Summary Report 2009-10

CISM Pilot Project Evaluation Report July 8, 2010

CISM Program Review Final Report 2012

CRA EAP - National Statement of Work Summer 2013 Department of Health Act

Devonish , D. Job demands, health and absenteeism: Does bullying make things worse? 2013

DM Dashboard OCISM Mar 2013

EAP 2012-2013 ANNUAL REVIEW AND PROGRAM PLAN Privy Council Office

Employee Assistance Services - Health Canada Revenue Authority Storyline

Everly, G.S.,  Davy J. A.,  Smith, Lating, J. M. and Nucifora,  F.C. Jr.    A defining aspect of health resilience in the workplace: A structural modeling approach.  Disaster Medicine and Public Health Preparedness. 2011.

Farahbaksah, S. A study of job burnout, sources and coping strategies in top managers of governmental organizations in province of Lorestan, Iran. Australian Journal of Basic and Applied Sciences. 2009

Health Canada Capacity Management Action Plan_EAS_Appendix A (FEB 08)

Health Canada Precis Re: The 2012-13 Telephonic Interview Results from EAP Clients of Employee Assistance Services (EAS) SHSD, RAPB 2013

Health Canada Precis re: The 2008-2009 Telephone Survey Results EAP-EAS 2009 through 2013-2014

Health Canada Precis Re: The 2013-2014 Telephone Survey Results EAP - EAS

ILA between Canadian Institutes of Health Research and Employee Assistance Services Regions and Programs Branch April 1, 2011 to March 31, 2014

July, 2012 Status on 2010 Audit of emergency Preparedness Psycho-social EPR, EAS

McTernan, W. P.,  Dollard, M. F.,  and  LaMontagne, A. D. . Depression in the workplace: An economic cost analysis of depression-related productivity loss attributable to job strain and bullying. Work and Stress. 2013

Merrill, R. M.,  Aldana S. G.,  Pope, J. E.,  Anderson, D. R.,  Coberley, C. R. Grossmeier J. J.,  and Whitmer, R. W.,  Self-rated job performance and absenteeism according to employee engagement, health behaviours, and physical health. Journal of Occupational and Environmental Medicine. 2013

MiniTracking_FY 08-09

MiniTracking_FY 09-10

MiniTracking FY 10-11

MiniTracking FY 11-12

MiniTracking FY -12-13

OCISM Annual Summary Report April 1, 2012 to March 31, 2013

Pitariu, H. D.,  and Budean, A.  The impact of occupational stress on job satisfaction and  counterproductive work behavior.  Human Resources Psychology, 2009

Policy Framework for the Management of Compensation

Psychological health and safety in the workplace -- Prevention, promotion, and guidance to staged implementation 2013

PSEPRT - Self-Assessment-Populated_Tool_March-28-2012

Psychological health and Safety in the Workplace - Standards

Report Audit of Emergency Prep Mar 2011

Robertson, I. T., Birch A. J.,  and Cooper, C. L.,  Job and work attitudes, engagement and employee performance: Where does phychological well-being fit in? Leadership and Organization Development Journal. 2011

Schaufeli, W. B.,  Bakker, A. B.,  and Van Rhenen, W. How changes in job demands and resources predict burnout, work engagement and sickness absenteeism.  Journal of Organizational Behaviour. 2009

Summary of Achievements CISM

TBS Policy Suite: Occupational Safety and Health

TBS Policy Suite: Policy on Employee Assistance Program

Treasury Board of Canada Secretariat - 2014-15 Report on Plans and Priorities

Westerlund, H.,  Nyberg, A.,  Bernin, P.,  Hyde, M.,  Oxenstierna, G.,  Jappinen, T.,  and Theorell, T.,  Managerial leadership is associated with employee stress, health, and sickness absence independently of the demand-control model. 2010

Wickramasinghe. V.,  Work -related dimensions and job-stress: The moderating effect of coping strategies. Stress and Health. 2010

Workplace Wellness and Productivity Strategy

Wood, S., and de Menezes, L. M.,  High involvement management, high performance work systems and well-being. International Journal of Human Resource Management. 2011

Appendix 2 - Logic Model

Appendix 2

Appendix 3 - Summary of Findings

Rating of Findings

Ratings have been provided to indicate the degree to which each evaluation issue and question have been addressed.

Relevance Rating Symbols and Significance:

A summary of Relevance ratings is presented in Table 1 below. A description of the Relevance Ratings Symbols and Significance can be found in the Legend.

Table 1: Relevance Rating Symbols and Significance
Issues Indicators Overall Rating Summary
Continued Need for the Program
What are the health/societal needs contributing to the need for the program?
  • Demonstration of health and/or societal needs (e.g. societal trends in mental health and impact on employment: absenteeism, disability plan, etc)
High Based on the information assessed in the course of this evaluation, there is a demonstrated need for interventions such those delivered by EAS.

The prevalence of mental health problems or illness, along with the recent changes and continued emphasis on efficiency which may have increased stress, suggest that there continues to be a need for interventions such as those delivered through EAS and to being more mindful of the psychological safety in the work environment, a factor known to enhance results of interventions such as EAP. The content of SOS could be enhanced by being adapted to educate managers on how they can help maintain a psychologically safe work environment. The adapted SOS sessions should also be delivered more broadly.

  • Program activities and reach connected to current needs
Partial
  • Views of stakeholders on program connection to needs
High
Alignment with Government Priorities
Do the objectives and priorities of EAS align with current Government of Canada priorities and departmental strategic outcomes?
  • Program objectives correspond to recent/current federal priorities
High The EAS is aligned with the Workplace Wellness and Productivity Strategy, a current priority of the Government of Canada.

EAS aligns with current government priorities to enhance the wellness and well-being of employees in order to increase productivity and reduce absenteeism. However, it is currently unclear what entity provides leadership on mental health within the federal core public service. At this point in time, the EAS appears to be positioned to contribute more directly to the strategic outcomes of the Treasury Board Secretariat, i.e., Government is well-managed and accountable, and resources are allocated to achieve results, than to a 'Health System Responsive to the needs of Canadians'.

  • Program objectives aligned with and contribute to departmental strategic outcomes
Partial
Alignment with Federal Roles and Responsibilities
What are the federal and Health Canada roles relating to the program area?
  • Program objectives aligned with federal jurisdiction
High As an employer, the federal government and Health Canada have clear legislated roles relating to the health, safety and well-being of federal employees. While other services provided across the government and within departments generally complement services offered by EAS, there may be overlap (similar services) of services for specific subsets of the target population, e.g., executives.

The federal government, and specifically Health Canada, have clear roles and responsibilities with respect to promoting and preserving the health (including the mental health) of its employees, as established in the: Department of Health Act; Treasury Board Policy on Occupational Safety and Health; Treasury Board Policy on Employee Assistance Program; as well as in various collective agreements.

  • Program objectives fit with departmental mandate and roles
High
Are there other programs that complement, overlap or duplicate the objectives of the program?
  • Presence/absence of other programs that complement or duplicate objectives of program
High

Legend - Relevance Rating Symbols and Significance:

High
There is a demonstrable need for program activities; there is a demonstrated link between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes; role and responsibilities for the federal government in delivering the program are clear.
Partial
There is a partial need for program activities; there is a some direct or indirect link between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes; role and responsibilities for the federal government in delivering the program are partially clear.
Low
There is no demonstrable need for program activities; there is no clear link between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes; role and responsibilities for the federal government in delivering the program have not clearly been articulated.

Performance Rating Symbols and Significance:

A summary of Performance ratings is presented in Table 2 below. A description of the Performance Ratings Symbols and Significance can be found in the Legend.

Table 2: Performance Rating Symbols and Significance
Issues Indicators Overall Rating Summary
Achievement of Expected Outcomes (Effectiveness)
To what extent has EAP progressed toward its immediate, intermediate and long-term outcomes?
  • Evidence of (performance data on) achievement of outputs and outcomes
Achieved EAP services provided by EAS seem to contribute to the increased use of resiliency skills and tools and thus contribute to an increase in productivity for the users remaining at work and to a modest reduction in absenteeism. The overall rating in the table above is exclusively associated with EAP performance given data is not sufficient to conclude on the extent to which SOS, PSEPR and OCISM contributed to attitude or behaviour change.

There is evidence indicating that EAS increased its client base significantly over the course of the evaluation period. The increase in the proportion of calls to Crisis and Referral Center were significantly lower than the increase in the consortium employee population, suggesting there may be an awareness gap in departments who recently joined the consortium and/or that the departments joined the consortium late in the fiscal year. EAP seems to contribute to the increased use of resiliency skills and tools to help federal employees in finding satisfaction at work. However, with respect to other EAS service lines, there is insufficient data to conclude as to whether those services contribute to attitude or behaviour change after the intervention.

  • Level of access to knowledge, tools and services
Achieved
Are EAP contributing to self-awareness and use of resiliency skills among its users?
  • Ability to deal with stressful circumstances
  • Capacity to put things into perspectives
  • Perceptions on one's sense of capability?
  • Ability to act positively when addressing difficulties
Achieved
Is there evidence to suggest that EAS contributes to an improvement of employees satisfaction and productivity?
  • Productivity is affected by emotional problems;
  • Productivity is affected by physical health
  • Change in health status
  • Self-reported absenteeism
Achieved
Demonstration of Economy and Efficiency
Have EAS undertaken their activities in the most economical and efficient manner?
  • Evidence of cost of producing outputs is as low as possible
  • Evidence of steps taken to enhance efficiency
  • Comparison of cost per outputs between similar programs
Achieved Based on the data assessed in this evaluation, EAS is delivered in an economic and efficient manner.

For the most part, EAS does demonstrate efficiency and economy. EAS deliver high quality services in accordance with industry and professional standards and has mechanisms in place to ensure economical use of public funds although updated technological infrastructures could contribute to greater efficiencies, e.g., for the collection of information and compensation of affiliate providers. When compared to one other jurisdiction, it appears that EAP fee per employee is in line with this comparator.

Is there appropriate performance measurement in place?
  • Existence of performance measurement framework or strategy
  • Adequate collection of performance information
  • Use of performance information in decision making
Progress Made; Further Work Warranted

Legend - Performance Rating Symbols and Significance:

Achieved
The intended outcomes or goals have been achieved or met.
Progress Made; Further Work Warranted
Considerable progress has been made to meet the intended outcomes or goals, but attention is still needed.
Little Progress; Priority for Attention
Little progress has been made to meet the intended outcomes or goals and attention is needed on a priority basis.
Table 3: Summary of Relevance and Performance Ratings
Evaluation Issue High Partial Low
Relevance:
Issue 1: Continued need for the program
What are the health/societal needs contributing to the need for the program? High N/A N/A
Issue 2: Aligned to federal government priorities
Do the objectives and priorities of EAS align with current Government of Canada priorities and departmental strategic outcomes? High N/A N/A
Issue 3: Program consistent with federal roles and responsibilities
What are the federal and Health Canada roles relating to the program area? High N/A N/A
Are there other programs that complement, overlap or duplicate the objectives of the program? High N/A N/A
Evaluation Issue Achieved Progress Made; Further Work Warranted Little Progress; Priority for Attention
Performance:
Issue 4: Achievement of intended outcomes (effectiveness)
To what extent has EAS progressed toward its immediate, intermediate and long-term outcomes? Achieved N/A N/A
Are EAS contributing to self-awareness and use of resiliency skills among its users? Achieved N/A N/A
Is there evidence to suggest that EAS contributes to an improvement of employees satisfaction and productivity? Achieved N/A N/A
Issue 5: Demonstrated economy and efficiency
Have EAS undertaken their activities in the most economical and efficient manner? Achieved N/A N/A
Is there appropriate performance measurement in place? N/A Progress Made N/A

List of Acronyms

ASE
Advisory Service for Executives
COA
Council on Accreditation
CRC
Crisis and Referral Center
DND
Department of National Defense
EAP
Employee Assistance Program
EAS
Employee Assistance Services
FNIHB
First Nations and Inuit Health Branch
JD-R
Job Demands and Resources
OCISM
Occupational Critical Incident Stress Management
PSEPR
Psycho-Social Emergency Preparedness and Response
PSHCP
Public Service Health Care Plan
RAPB
Regions and Programs Bureau
RCMP
Royal Canadian Mountain Police
SHSD
Specialized Health Services Directorate
SOS
Specialized Organizational Services
VAC
Veterans Affairs Canada
VNR
Vote-Netting of Revenues

Endnotes

Footnotes

Page details

Date modified: