Language selection

Government of Canada / Gouvernement du Canada

Search

Evaluation of the Workplace Information and Research Activities of Employment and Social Development Canada Labour Program

Executive summary

The Workplace Information and Research Division Footnote 1 of ESDC’s Labour Program is responsible for the management of the Workplace Information and Research Activities (WIRA) of the Labour Program. These activities include the collection, analysis and provision of information on collective agreements to various users, including collective agreement negotiating parties in Canada. As part of its activities, the division also conducts other related research activities for internal, as well as external users to inform labour policy development, program reviews and other decision-making processes. Footnote 2

During 2010-2011 to 2012-2013, the division underwent significant organizational changes to its publication and data collection processes as part of the Business Process Redesign. Footnote 3 Those changes prompted the Labour Program to request an evaluation of their impacts on the effectiveness and efficiency of WIRA. This is the first evaluation of the Labour Program’s information and research activities.

The main objectives of the evaluation were to assess the impacts of the Business Process Redesign changes and the relevance, continued need, and the effectiveness and efficiency of WIRA operations. The evaluation is unique in that it provides evidence on the quality of activities, users’ satisfaction and efficiency of the WIRA operations as a federal departmental service to its users, both internal and external, rather than as a traditional program with immediate and long-term outcomes. This report presents the findings and recommendations of the WIRA evaluation covering the period from 2008 to 2013.

Evaluation data collection activities took place between September and December 2014. Appendix B provides a detailed description of the methodologies used, and their strengths and limitations.

Key findings

  • Continued need: The evaluation confirmed that WIRA products and services are used by a wide range of both internal and external users and known and unknown users. Users of WIRA products and services reported they use the products to plan, negotiate and settle collective agreements, as well as labour disputes. They also use them to conduct research in the area of labour relations. WIRA research products are used to develop policies and support decision making processes. The division contributes to ESDC’s mandate relating to the collection, analysis and dissemination of data and information on labour matters in both federal and provincial labour jurisdictions, as well as its international reporting obligations. Users reported that they will likely use WIRA products in the future and that their demand will increase, although some users indicated that new products or a broadened scope of existing products and improved accessibility are needed.
  • Alignment of WIRA with ESDC mandate: WIRA products and services contribute to ESDC’s Strategic Outcome 2: “Safe, fair and productive workplaces and cooperative workplace relations.” By supporting negotiating parties in settling collective agreements, WIRA products contribute to fair and cooperative workplace relations. The Treasury Board Secretariat (TBS), Statistics Canada, and the Bank of Canada use WIRA products and services to prepare briefing notes and policy options for senior management to support decision making. For example, economic impact analyses, conducted by the division, help the Government decide on proposed amendments to the Canada Labour Code.
  • The Workplace Information and Research Division is the unique national source of information on collective agreements: The division maintains information on collective agreements in federal and provincial jurisdictions. Thus, the division is the only source of information on collective agreements with a national focus. Provinces and territories, as well as major unions maintain and provide information on collective agreements, but only for their respective jurisdictions. In addition, some WIRA products, such as information on union membership and affiliation, and work stoppages at the national level are unique to the division.
  • The Workplace Information and Research Division needs to better match its products to users’ needs to be efficient in its operations: The Performance Measurement Strategy (PM Strategy) for WIRA noted that the division produces and makes its products available on its website but does not actively promote them. The PM Strategy recommended that the division collect performance data to better understand and meet users’ needs efficiently. The document review suggests documenting and describing the theory of change Footnote 4 underlying WIRA to understand the causal link between its activities and outputs or outcomes to ensure that better performance measures are selected and appropriate performance information/data are collected. Accurate user contact information and regular communication to seek users’ feedback on the quality of products and services provided are necessary to match products with users’ needs.
  • The Workplace Information and Research Division is effective in making high quality products and services easily accessible to its users: Most users indicated that the access to WIRA products and services is easy and that they received help from the division easily and quickly. However, navigability of Negotech, the repository of all collective agreements, and the division website and awareness of new products were highlighted by some users as challenges to accessing WIRA products. WIRA products are described by most users as accurate, valid, useful, comprehensive and readily available for their use. However, a small group of users who were less satisfied with the comprehensiveness of products indicated that the division could make more collective agreements (i.e., for small unions representing less than 100 employees) available and could conduct more analyses by region, occupation, industry, and unions to increase their usage. Users who were less satisfied with the timeliness indicated that sometimes answers to their urgent and complex requests were received too late, rendering the information no longer useful.
  • Impacts of changes made since 2010 were positive on WIRA operations: In 2011, WID and RDD merged into one single division, the Workplace Information and Research Division to manage WIRA operations. These changes were implemented in an effort to identify challenges and suggest ways to improve WIRA operations. This resulted in more WIRA products offered online increasing their accessibility. The changes also resulted in a reduction in the number of requests by e-mail, as well as in time and effort of staff involved in data collection and publication processes.

Recommendations

To ensure that WIRA products and services continue to be meaningful for new, potential and regular users, it is recommended that:

  1. The Workplace Information and Research Division refines its engagement strategy to increase its client base and enhance awareness of WIRA products and services;
  2. The Workplace Information and Research enhances the online user experience by improving the navigability and the search capabilities of online databases and the division’s website;
  3. The Workplace Information and Research explores ways to improve the speed of processing users’ requests to ensure that information is available in time to be useful for its internal and external users; and
  4. The Workplace Information and Research considers implementing its new Performance Measurement Strategy to collect appropriate data to better understand and serve the needs of users.

Management response

The Workplace Information and Research Division refines its engagement strategy to increase its client base and enhance awareness of WIRA products and services.

The division agrees with this recommendation.

Actions:

A weekly survey of all individuals who have used WIRA products and services has been in place since January 2015 to collect information about users’ satisfaction with WIRA products and services, as well as suggestions for improvements. A monthly dashboard is produced for monitoring by the division’s management.

Planned activities:

  • The division will ensure that contact information for users who make requests in-person or by email or phone is collected and entered correctly into the database. This will be an on-going practice.
  • The division will engage in consultations with ESDC partner branches (Innovation, Information and Technology Branch (IITB), Service Canada’s Citizen Service Branch (CSB), Privacy, and Legal Services) to explore the possibility of adding an online survey to the division’s website to collect contact information and gather users’ feedback on WIRA products and services online. The consultation will take place over the summer of 2015 with the objective of having the survey in place before the end of fiscal year 2015-2016.
  • The division will collaborate with the Public Affairs and Stakeholders Relation Branch (PASRB) to undertake communication initiatives to increase awareness, such as:
Twitter:
  • PASRB will send Tweets from the Labour Program Twitter account to improve the division’s visibility and highlight the availability of new products and services developed by the division with links to those products. This will be implemented on a case-by-case basis following final approval of the Management Response.
  • Similarly, Tweets containing quick facts and statistics extracted from the division’s publications will be used to promote the publications and may be used in cross‑promotion for Ministerial events or other Labour Program initiatives. This will be implemented following final approval of the Management Response.
Web feature:
  • A web feature will be developed for use on the Labour Program homepage to provide a link to new publications developed by the division to increase the visibility and accessibility of the division’s publications. This will be implemented following final approval of the Management Response.
Google Analytics:
  • Will be used to analyze visitor traffic and provide insight into how visitors use the division’s web pages.
  • Key words/common search terms, aligned with publication titles, will be used in future publications to ensure that the division’s publications are among the top search results lists. This will be completed by August 2015 and then done on an ongoing basis.
  • The “Common Questions” section of the Labour Program’s website will direct users looking for industrial relations information to the division’s Contact page. This will be implemented upon final approval of the Management Response.
  • The division will communicate with other organizations and professional associations who could potentially use its products and services, such as the Canadian Industrial Relations Association, Brock Labour Studies, and the Conseil du Patronat du Québec, to inform them about its products as well as to explore the possibility of having a direct link for the division’s online products on their respective websites. This will be an ongoing practice.
  • The division will implement an internal research newsletter to be sent out to Labour Program executives, policy analysts and others to be determined. The newsletter, to be launched by July 2015, will include a short summary of recently completed research reports, along with a division’s Share Point link to those documents.
  • The division will explore the possibility of having a link on the Labour Program intranet that will direct users to the division’s product page.
  • The division, in consultation with the Federal Mediation and Conciliation Service, will examine the feasibility of expanding the current thresholds for monitoring and analyzing collective agreements in both the federal and provincial jurisdictions and determine the need for this information and the resources required to expand the thresholds. This will be completed by December 2015.

The Workplace Information and Research Division enhances the online user experience by improving the navigability and the search capabilities of online databases and the division’s website.

The division agrees with this recommendation. It will need active support from IITB, CSB and PASRB to improve the navigability of the Labour Program (division) website.

Planned activities:

  • The division will engage in consultations with the IITB, CSB and PASRB, over the summer of 2015, to explore possible improvements to the usability of the Labour Program website, keeping in mind the upcoming migration of Labour Program content to Canada.ca.
  • Subject to the availability of resources, the division will conduct a usability assessment of its online database Negotech to identify ways to enhance user experience. This will be completed by March 31, 2016.
  • As the division also releases its data through Statistics Canada, users access division’s product through the Canadian Socio-economic Information Management system (CANSIM) as well. The division will assess and redesign where necessary the format and content of work stoppages and wage data being sent to CANSIM. This will be completed by October 2015.

The Workplace Information and Research Division explores ways to improve the speed of processing users’ requests to ensure that information is available in time to be useful for its internal and external users.

The division agrees with this recommendation. Currently the majority of simple requests are responded to within the expected time frame of two business days. Timeliness refers to the elapsed time either to respond to user requests, or between expected and actual date for posting publications online. This timeliness is already being tracked and documented for each client request (ICAMS, since 2008) or for each publication (Divisional Application Toolbox (DAT) /Research, Analysis and Publication (RAP)) – Publication Status, since March 2015).

Actions:

  • Users making complex requests are always consulted and advised on the expected timelines and kept apprised of any changes.
  • The division already has in place a number of measures to improve request processing time. Given current division’s capacity, further improvements in request processing time will require devoting more resources.

Planned activities:

  • The division will continue to monitor its ability to respond to requests in a timely manner based on the data captured by these applications (ICAMS and DAT) and the implementation of the Performance Management Strategy. This activity will be completed by March 2016.
  • The division will collect data on performance indicators to assess the quality, usefulness and timeliness of research reports using a short survey of internal requestors of complex research. Data collection and analysis will be completed by March 2016.
  • The division will begin tracking the time spent on the more complex research projects using a new Status Report to document the timelines of all projects undertaken by the division. The Status Report will be in place by May 2015.
  • The division will consult with PASRB to explore ways to further improve the timeliness of its online publications and also the possibility of establishing consistent publication schedules, as well as the identification of next steps. This consultation will take place during 2015-2016.

The Workplace Information and Research Division considers implementing its new Performance Measurement Strategy to collect appropriate data to better understand and serve the needs of users.

The division agrees with this recommendation.

Actions:

  • The division developed a Performance Measurement Strategy in April 2014. A number of performance indicators were identified to assess the performance of the division.

Planned activities:

  • The division will document and describe how the division’s activities effectively contribute to the division’s outputs and outcomes. This will be an on-going activity starting in September 2015.
  • The division will develop an appropriate performance data collection system by July 2015
  • On a continuous basis, data on performance indicators will be collected and analyzed to ensure they are accurate and relevant and will be used to improve the management of the division. The report on the first analysis will be completed by December 2015.

Introduction

1.1 Evaluation context

This evaluation of the Workplace Information and Research Activities (WIRA) was undertaken as part of the ESDC Evaluation Plan 2013-2014 to 2017-2018 and with respect to the 2009 Treasury Board Policy on Evaluation. The program had not been previously evaluated in its 40 years of operation.

During the period from 2010-2011 to 2012-2013, the Workplace Information and Research Division made significant operational changes in its publication and data collection processes to increase efficiency as a result of the Business Process Redesign. The changes to WIRA operations prompted the necessity to conduct this current evaluation to understand if those changes were appropriate in helping to achieve expected results.

1.2 Description of the Workplace Information and Research Division

The Minister of Employment and Social Development Canada is mandated to collect, analyze, interpret, publish and disseminate information relating to human resources and skills development or to social development, as stipulated by the article 6(a) of the Employment and Social Development Act. This includes data and information on labour relations in Canada.

ESDC’s Labour Program is responsible for the collection, analysis and dissemination of data and information on labour matters in both federal and provincial labour jurisdictions. These activities were performed by the Workplace Information Division (WID) which existed within Labour Program for more than 40 years until 2010. In 2007, the Research and Data Development unit (RDD) started conducting research activities including analysis and research on emerging issues, programs and policies to support the Labour Program mandate. In 2010-2011, WID and RDD merged into one single division, the Workplace Information and Research Division (the division), currently responsible for the Workplace Information and Research Activities (WIRA) in the Labour Program.

The division collects, analyzes and disseminates information on collective bargaining in both federal and provincial jurisdictions. The division collects information on collective bargaining processes and agreements, work stoppages, as well as on Canadian labour organizations’ membership and affiliations. The division has both internal and external users of its WIRA products and services. Internal users include Labour Program groups, senior management and Ministers. External clients include TBS, Statistics Canada, the Bank of Canada, unions, employers, researchers, lawyers, consultants, media and the general public, as well as the International Labour Organization. WIRA products and services are used by negotiating parties to facilitate collective bargaining processes, by government officials for policy development and decision making, and by researchers for their work in the area of labour relations. The division also helps to meet the government’s international obligations by providing reports on work stoppages to the International Labour Organization on an annual basis. In addition, information on wage adjustments, work stoppages and wage projections is shared with TBS, Statistics Canada and the Bank of Canada.

The division conducts research and econometric/statistical analyses related to collective bargaining and all other areas of the Labour Program mandate, such as labour standards, occupational health and safety, employment and pay. The research and analyses produced by the division are used by the Labour Program for policy development, by the Department of Finance and the Bank of Canada to monitor macroeconomic conditions, and by TBS to monitor collective bargaining and negotiations. In addition, the division supports TBS and ESDC in the analysis of the economic impact of various labour issues.

The budget (operations and salaries) for WIRA has been declining from $3.3 million in fiscal year 2009-2010 to $ 2.4 million in 2012-2013.

2. Evaluation scope, questions and methodology

The evaluation assessed the relevance and continued need as well as the effectiveness and efficiency of WIRA. The evaluation focused on identifying WIRA products and services and determining who the various users of WIRA products and services are. The evaluation also examined the quality and the accessibility of WIRA products and services, as well as their impacts on collective bargaining and labour policy development processes.

The six evaluation questions examined as part of this evaluation are as follows:

  • To what extent are WIRA products (information, analyses and research reports) accurate, reliable and relevant to types of users (Unions, employers, researchers, Labour Program managers, etc.)? 
  • To what extent are WIRA aligned with ESDC and the government-wide priorities and jurisdiction?
  • To what extent are the WIRA duplicating or complementing or enabling other ESDC programs, initiatives or functions?
  • To what extent are WIRA products accessible to different types of users (unions, employers, researchers, labour program managers, etc.) in a timely manner?
  • What are the impacts of WIRA products (information, analyses and research reports) on the dialogue and negotiations related to collective bargaining and on policy development in the Labour Program?
  • To what extent has the business process redesign improved the efficiency of the core operational processes, enhanced services to users, and prepared WIRA for open data?

The evaluation covered the division’s research activities and operations during the fiscal years, 2008-2009 to 2012-2013.

The methodology used five lines of evidence:

  • Document review;
  • Administrative data analysis;
  • Key informant interviews with Labour Program representatives;
  • Survey of users of WIRA products and services; and
  • Surveys of potential users of WIRA products and services.

Detailed information on the evaluation methodologies and their strengths and limitations are found in Appendix B.

3. Key findings

This section presents key findings from all lines of evidence used in this evaluation. The report is structured by strategic insights and findings. Footnote 5

3.1 Relevance

All lines of evidence demonstrated the importance and continued relevance of WIRA products and services in supporting collective bargaining, policy development and decision-making processes related to labour relations in Canada.

WIRA products and services are used by a wide range of users.

Analyses of the division’s administrative data (ICAMS Footnote 6 data) indicated that those who requested WIRA products and services identified themselves as employers, individual workers, government representatives (federal, provincial/territorial and municipal), unions, lawyers, researchers, universities/colleges, students, consultants, media and the general public. Footnote 7 These categories of users are consistent with the job descriptions of respondents to the User Survey as shown in Figure 1. Footnote 8

Based on this survey, public servants (26%) researchers (16%) individual workers (16%) and employers (13%) are the predominant users of WIRA products and services. Footnote 9 , Footnote 10 This evidence was also supported by the Key Informant Interviews. Footnote 11

Figure 1: Respondents/Users by Job Description

Text description

Figure 1 presents the User Survey Results on the percentage of the respondents/ users type by job description. The figure presents the percentage of 10 respondents/users types of job descriptions. From the top to the bottom, the respondent job descriptions are: public servant (excluding negotiator/conciliator), researcher, individual worker, employer, union official, negotiator/ conciliator, consultant, lawyer, general public and student.

Source: User Survey, 2014, n=210

* Excludes negotiator/conciliator

WIRA products and services support collective agreement negotiations, policy development and decision-making processes.

A high percentage of surveyed users perceived WIRA products as useful in supporting collective agreement negotiations (97%), research (97%), labour policy development (92%) and labour dispute settlement (84%). Footnote 12 A significant proportion of non-government users (53%) Footnote 13 compared to government users (23%) reported that WIRA products and services helped in planning, negotiating and settling collective agreements. Footnote 14 In addition, 45% of users also reported that WIRA products are used for planning, negotiating and settling labour relations disputes. Footnote 15

Surveyed users who provided information on the importance of WIRA products in collective bargaining processes ranked Collective Agreements Footnote 16 (99%), Wage Adjustments in Collective Agreements (94%) and Benefits and Working Conditions in Collective Agreements (93%) as the most important WIRA products. Footnote 17

TBS requests information on wage adjustments and wage projections for its own reporting and to prepare for negotiations with public servants, and uses information on WIRA wage projections to adjust the salaries of Members of Parliament. The Bank of Canada uses information on wage projections as an economic indicator. Footnote 18

The key informant interviewees reported that WIRA products are used by collective agreement negotiating parties to prepare their respective positions at the negotiation table. Footnote 19 They also indicated that data on collective agreements and labour organizations, as well as research reports and econometric analyses are used in preparing policy options that support policy development and decision-making processes. They reported that WIRA products support the Minister and senior managers in their decision-making processes on labour matters. Footnote 20

Key informants also reported that the government uses WIRA research when considering policy options to address work stoppages. Internal users of WIRA products perceived the division as the “go-to-place” for reliable information in supporting policy development. Footnote 21

WIRA products respond to current and future information needs

Findings from the User Survey indicated that, on average, each WIRA product was used by individual respondents between two and four times over the last 12 past months. Footnote 22 , Footnote 23 Collective agreements are the most frequently requested products. On average, each user accessed them 11 times in the last 12 months. Public servants, union officials, negotiators, lawyers and researchers were the heavy users of information on collective agreements. Footnote 24 The administrative data indicated that Negotech and collective agreements were the most requested products from 2008 to 2013. Demand for Negotech increased from 10.8% of total requests in 2008 to 23.2% in 2013, and the demand for collective agreements increased from 6.2% of total requests to 20.4% in the same time period. Footnote 25

The majority of surveyed users (88%) who provided information on the need of WIRA products indicated that they will likely continue to use new and existing WIRA products and services in the future. Footnote 26

Most Labour Program representatives who are frequent users of WIRA products and services (including research reports) indicated that their future needs of WIRA products will increase. The capability of the division’s research group to produce high quality products is acknowledged within the department, and the division has started to solicit research ideas from other groups in the Labour Program. As a result, the number of requests for research products has increased from a wide variety of internal federal government users, including those in the Labour policy group, Federal Mediation and Conciliation Service, Federal Compensation, Employment Equity, Occupational Health and Safety, and Labour Standards. Footnote 27

TBS and the Bank of Canada request information on collective agreements, including the information on wage adjustments on a monthly basis and wage projections on a yearly basis. The division also shares data on work stoppages with Statistics Canada on a quarterly basis. Footnote 28 These requests are ongoing and there is no indication that the demand for WIRA products by these organizations will change in the near future.

WIRA products and services support the ESDC mandate and Strategic Outcome 2: “Safe, fair and productive workplace and cooperative workplace relations.”

WIRA align with the ESDC Strategic Outcome 2: “Safe, fair and productive workplaces and cooperative workplace relations” as reflected in the ESDC Program Alignment Architectures for 2012-2013 and 2013-2014. Footnote 29 WIRA products on collective agreement processes also contribute to the promotion and sustainability of cooperative workplace relations within the federal jurisdiction and beyond (Departmental Performance Reports 2012-2013). As per the Canada Labour Code, Section 115, the division helps to fulfill ESDC’s regulatory mandate to collect and disseminate information on collective agreements.

WIRA products and services are aligned with the Federal Government’s interest/priorities

Key informant interviewees were unanimous that WIRA products and services are well aligned with ESDC and government-wide priorities. Users indicated that WIRA products are used for planning, negotiating and settling collective agreements. Footnote 30 The document review revealed that economic impact studies are the most common research reports prepared by the division. These studies are conducted to assess the impact that workers’ actions, employers’ decisions and government legislation may have on the productivity and economic interests of businesses, industry and Canada as a whole. WIRA research products are prepared primarily to support Labour Program and Government of Canada decision makers in the domain of labour relations and all related issues to the Canada Labour Code to ensure good working relations and prevent loss of productivity. Supporting this, key informants indicated that WIRA products are used to prepare presentations and briefing notes for senior managers, including ministers, on different labour files. WIRA research and other products also assist TBS in implementing the government's priority of preventing loss of productivity. Footnote 31

WIRA support the Government of Canada to fulfill its international reporting commitments.

The division helps ESDC to meet the government’s international reporting obligations and respond to requests on labour matters from the International Labour Organization, the Organization for Economic Co-operation and Development and other countries. The document review indicated that, on an annual basis, the division reports Canadian nation-wide work stoppages information to the International Labour Organization. The work stoppages information that Statistics Canada publishes comes from the division, which is the unique source of information on work stoppages at the national level in Canada. At the provincial/territorial level, only three provinces −British Columbia, Alberta and Ontario− offer information on work stoppages similar to WIRA products. However, their information focuses on activities within their respective jurisdictions, and therefore lacks the national perspective required to fulfill Canadian international obligations. Footnote 32 In addition, the division has no current working arrangements with Provinces and Territories to share information. The division may consider collaborating with at least the above three provinces and other major labour organizations, to share information and analysis of mutual interest.

WIRA products and services play an enabling function for other Labour Program groups

The value of WIRA products and services is rooted not just in its federal focus, but also in its focus on the Labour Program. The word “enabling” was repeatedly used by key informants to describe the impact WIRA have on other Labour Program groups. Many frequent internal users felt that their work was dependent upon WIRA information and products.

Users of research products from other Labour Program groups consider that their jobs would be difficult without the help from the WIRA research group. For those users, no other research groups in ESDC offer the same type and quality of research and data pertaining to the Labour Program mandate. Footnote 33

This enabling role is supported by the User Survey that indicated that a high percentage of public servants (26%) are users of WIRA products. Footnote 34 The more frequent users, especially the public servants, use each WIRA product 18 times a year on average. Footnote 35

The division acts as a single repository of collective agreements and related information in Canada. This does not duplicate the activities of other federal departments or provinces and territories in this regard.

Findings from the document review indicated that the online database, Negotech, maintained by the division, contained 44,094 collective agreements collected from both public and private sectors in Canada. Footnote 36 Negotech contains full text and summaries of collective agreements dating back to 1976, which are accessible by the general public. Footnote 37 Although copies of some collective agreements are available from a variety of other sources, the division is identified as the single repository of the majority of collective agreements in Canada. It is important to mention that provinces and territories, large public sector employers such as universities or hospitals, provide information on their collective agreements and cite Negotech and ESDC among their sources of information. The province of Manitoba provides a direct link to WIRA products as a source of national and provincial information. Other online sources include: large public sector employers such as universities or hospitals; major unions and labour organizations. Footnote 38

Information on collective agreements available on provincial websites varies by province, and rarely dates back prior to 2002. Three provinces −British Columbia, Alberta and Ontario− provide some basic data and analyses such as quarterly wage settlements, wage settlement summaries, status of major negotiations and work stoppages, similar to WIRA products, but only since 2012. Both the Conference Board of Canada and Statistics Canada were mentioned as alternative sources of information but their focus is limited to topics of specific nature. Footnote 39

It is important to mention that the collective agreements maintained by the division are limited to unions that shared Footnote 40 information with the division and that represent 100 employees or more in the federal jurisdiction, and 500 employees or more in the private sector. Provinces and territories and other labour organizations maintain information on collective agreements for unions similar to those covered by the division, and also for smaller unions in their jurisdictions. In addition to the issue of coverage, some organizations want to conduct different types of analyses than what the division is currently providing. This is consistent with responses of some users. For example, 44 respondents to the User Survey noted why they were unlikely to use WIRA products and services in the future: less than 66% indicated the need for new products or broadened information (i.e., on small unions) and 9% wanted more detailed information on existing products. Footnote 41

3.2 Effectiveness

The division produces accurate, valid, useful, comprehensive and timely information that has an impact on collective bargaining and labour related policy development and decision-making processes. Users surveyed reported that WIRA products are used in planning, negotiating and settling collective agreements and labour disputes.

The majority of users assessed the WIRA products and services as accurate, valid, and useful. However, the levels of satisfaction with timeliness and comprehensiveness were lower.

Key informant interviewees reported that the process by which information on collective agreements is collected and prepared ensures its accuracy. To achieve this, the division maintains an up-to-date calendar of collective agreements to determine when new data will be available. Information is collected directly from unions, ensuring its accuracy and completeness and it is verified before being released to the public. Footnote 42 As a result of this process, internal and external users of WIRA products were satisfied with data accuracy. Footnote 43 , Footnote 44

The division’s research capabilities are regarded as very valuable by Labour Program groups as well as other federal departments, agencies and crown corporations. These organizations need the division’s expertise to help in the preparation of policy papers and ministerial briefing notes. The division also receives requests to prepare costing for policy options and impacts related to Labour relations issues.

To ensure that research planning takes policy needs into consideration, a Director-General level consultation process with other Labour Program groups has been established. Several Labour Program users commented positively on the initiative. It is seen as an important and positive tool to help the division tailor its research (and products) to its users’ needs. The key informant interviewees suggested that information shared at the Director-General level also needs to be shared between Directors and filtered down. Footnote 45 This suggests that similar initiatives to consult other users, including federal departments, agencies, and crown corporations, as well as external users, should be explored to ensure WIRA products continue to meet their needs.

The users of WIRA products and services expressed a high level of satisfaction regarding quality in general, but their satisfaction was lower in terms of product comprehensiveness and timeliness. The majority of respondents to the User Survey noted that WIRA products are accurate (86%), valid (88%), and useful (86%), but satisfaction was lower for comprehensiveness (78%) and timeliness (69%). Government and non-government users of WIRA products also reported a lower level of satisfaction for comprehensiveness and timeliness compared to their level of satisfaction with accuracy, validity and usefulness of WIRA products. Footnote 46

Comprehensiveness

The low level of user satisfaction related to the comprehensiveness of WIRA products and services is possibly explained by the current division’s union size coverage. This is consistent with what current users would like to see as new division’s products. Their suggestions indicated the need for new products or a broadened scope of current products to include smaller unions and detailed analyses. Footnote 47 The three reasons why potential users don’t use WIRA products and services were that they don’t need them (45%); they do their own research (14%); and they are unsure why they don’t use WIRA products and services (14%).

Among potential users, those who reported that WIRA products are not useful to them mentioned lack of information on small size unions as one of their reasons. Footnote 48 One point raised at the Extended Evaluation Working group meeting was that senior management requests for information on small unions (i.e., unions of 50 employees and less) are not covered by WIRA operations. Footnote 49 This suggests that information on small unions would meet a certain number of users’ needs.

Timeliness

The low satisfaction level among users surveyed about the timeliness may be explained by the following:

  • the user’s waiting time for information requested;
  • the time elapsed between the expected date of publication and when the information was finally published; and
  • the time it took to find information when the user was searching on the division’s website or database.

Thirty-one per cent of respondents to the User Survey required assistance in accessing the information they requested. The majority of these respondents (80%) found it easy to obtain the necessary help from the division. Satisfaction with the timeliness of the help received among its users is very high (88%). Dissatisfaction rises sharply when respondents have to wait more than seven days for assistance. Footnote 50

Most key informants reported that when they needed information from the division, they generally received it within their specified timeframe. However, some key informants reported that for certain requests, the information was provided too late and was therefore no longer useful. The division’s managers also indicated that this happens for special requests that require more research and more time to produce the requested information.

The new PM Strategy highlights the importance of monitoring how the division meets its targeted deadlines for its publications and responses to users’ requests. Footnote 51 This suggests that the division acknowledges the importance of the collection of accurate performance data to meet users’ information needs in a timely manner.

WIRA products and services are useful in supporting the dialogue and negotiations related to collective bargaining and policy development processes.

Key informant interviews indicated that the data and statistics on collective agreements in WIRA products are helpful in collective agreement negotiations. Footnote 52 This is consistent with the User Survey results that indicated that WIRA products assist negotiating parties to plan and develop their positions for negotiations.

The results of the User Survey indicated that WIRA products and services are used extensively by labour unions (71%) for planning, negotiating and settling collective agreements and labour disputes. Footnote 53 A large majority of respondents felt that WIRA products and services were useful for collective agreement negotiations (97%); research (97%); labour policy development (92%); settling labour disputes (84%); and program and policy reviews (82%).

The results of the User Survey also indicated that almost all WIRA products and services were perceived as useful. Their use in collective agreement negotiation and in research was seen as useful by nearly all users (97%). Compared to employers (59%), a significant proportion of users from government (85%), labour organizations (81%) and universities/colleges (100%) viewed the research reports as very useful. Footnote 54

Collective Agreements (99%), Wage Adjustments in Collective Agreements (94%) and Benefits and Working Conditions in Collective Agreements (93%) were the products that users felt were most important in supporting collective agreement negotiations. Footnote 55

The division conducts various types of activities to produce research reports. These reports differ in scope, depth and levels of analysis, and can range from a briefing note of a few pages to a major research paper of 20 or more pages. The division also conducts literature reviews, prepares descriptive statistics and generates statistical models as required by a project. These reports/papers address topics related to the Labour Program’s mandate such as occupational health and safety (i.e., rates of workplace injury, rates of industry compliance with occupational health and safety standards), unionization (i.e., union coverage rates, effectiveness of unions), worker rights and wage issues, and other outstanding issues such as the impacts of new developments (i.e., back-to-work legislation). As well, they inform policy development in the above mentioned areas.

Economic impact analyses are the most common research reports prepared by the division. These analyses assess the impact that actions of workers, the decisions of employers and government legislation may have on the productivity and economic interests of businesses, industry and Canada as a whole. The division also conducts detailed analyses of proposed or implemented actions and makes projections about possible outcomes (i.e., wage projections) to inform decision-making. Footnote 56

Some users of WIRA products are unaware that the products come from the division.

Results from the Potential User Survey showed that a minority of potential users Footnote 57 were aware of the division (18%) or activities of the Labour Program supporting collective bargaining processes (22%). Despite this low rate of awareness, when potential users were asked if WIRA products and services met their needs, the response was positive for the majority of users (73%). Only 22 of the 101 respondents indicated that they had not used WIRA products and services. Footnote 58 This suggests that potential users who used WIRA products in the past and ceased to be users are not aware of recent developments in WIRA products or were not aware that what they used were WIRA products.

The Internet is the preferred way to access WIRA Products

In 2013, the most common means of accessing WIRA products was through the Internet (48%) and telephone requests (34%) Footnote 59 . Results from the User Survey revealed that more than 80% of users accessed WIRA products online. A majority of these users prefer to continue to access most WIRA products online. For example, 91% of users expressed their preference to access collective agreements online. This was also true of more than 80% of users who access other WIRA products online. The User Survey revealed that there are still users who preferred to access WIRA products and services by e-mail (10%) or telephone (3%) requests. Footnote 60 This is expected, as there will always be some users who need additional information or guidance on the products they use or help in finding and using other WIRA products.

3.3 Efficiency and economy

In 2010, the division management undertook changes to WIRA operations to gain efficiencies. These changes included:

  • the redesign of the division’s website and its upgrade to HTML5;
  • the formatting of the web pages from multiple page to single page format;
  • the automation of data collection and publication processes;
  • the posting of more WIRA products online; and
  • the removal of some products from the website such as historical data because efforts to update them to comply with the federal guidelines would have been resource intensive. Footnote 61

Findings from different lines of evidence demonstrated that changes undertaken in 2010, including placing more WIRA products online, resulted in efficiency gains.

Text description

The Figure 2 presents information on how the different methods used to access WIRA products and services evolved between 2008 and 2013. The different methods used by various users are: email, fax, internet, mail, personal visits, telephone and voicemail. Since 2009, the access by email started dropping while the access through internet started rising.

Source: ICAMS data

The decision to place more products online resulted in increased access to WIRA products and services through the Internet and a decreased number of requests by e-mail.

The administrative data analysis indicated that the use of the Internet to make requests for WIRA products and services has increased since 2010. The increase in Internet-based requests corresponds with more WIRA products being published and made accessible online since 2010 and a corresponding decline of requests by e-mail and by telephone (Figure 2). Telephone requests have been declining since 2011, but they still form a substantial portion of the requests received for WIRA products and services. Footnote 62 Currently, most telephone calls seek information on the availability of an existing or new product or request customized tables or a specific product not available online rather than to accessing regular online WIRA products. Footnote 63

The users of WIRA products and services have increasingly moved to online access but some are still contacting the division by phone or email. Certain users emphasized that this is necessary to provide and seek clarity on ad-hoc and complex requests. It may be worthwhile to anticipate a need for additional staff to respond to possible increased numbers of ad-hoc and more complex requests.

Changes made since 2010 to WIRA operations have positively impacted the quality and accessibility of WIRA products, as well as on WIRA operations.

The division is contributing to the Government of Canada’s “Blueprint 2020” and “High Performing Organization” agenda by finding ways to process products and requests more efficiently. For example, a backlog in the preparation of the summaries of collective agreements was outstanding for three years. By making changes to streamline its business operations, the division was able to clear this backlog in three months without increasing its level of resources. Footnote 64

Findings from the User Survey indicated that the net impact of the changes to WIRA in 2010 is positive. For example, the majority of users (85%) who used WIRA products before and after 2010 reported that product quality improved (40%) or stayed the same (45%) after 2010 and 81% indicated that the changes had positive impact on accessibility. Footnote 65

Between 2009 and 2013, a significant increase of referrals Footnote 66 and call backs Footnote 67 was observed (Figure 3). Footnote 68 These enquiries could be the result of offering more WIRA products online during the same period, as users may not have understood the changes or the online products. Monitoring the progress of each action taken and its impact is important to ensure that corrective actions are taken appropriately. This suggests that there is a need to improve communication with users about changes being made to WIRA operations and their implications in advance. Communication with users requires that the division maintains up-to-date user contact information. The analysis of the contact information provided by the division as a sampling frame for the User Survey revealed that majority of contact information was either invalid or duplicates. Of 5,965 contact information sets provided by the division, only 17 % (992) represented the actual number of unique users. Footnote 69

Text description

N/A

Source: ICAMS Client by Product, 2014

Data collection automation

The document review indicated that the division management has implemented changes to business operations to increase WIRA efficiency. For example, the automation of the HTML Footnote 70 transformation process allowed some savings in time, while increasing the volume of data being posted online. The collection of information on collective agreements was improved with the introduction of automated e-mails sent to contacts. Footnote 71 It also improved contact information tracking. All operational changes taken together resulted in fewer staff interventions and approvals, which led to information being uploaded to the Labour website faster, and collective agreement summaries being uploaded on Negotech in a more timely fashion. Footnote 72

Development of the Performance Measurement Strategy (PM Strategy)

The document review indicated that the 2014 PM Strategy requires the monitoring of publication timelines to ensure that reports, data tables, and other products are completed and published on time. The PM Strategy recommends the inclusion of a client feedback tool to continuously assess regular user satisfaction. Footnote 73 This is consistent with the general opinion, among the division’s managers interviewed, that the new PM Strategy will ensure that the changes made within the division will help to achieve the intended efficiencies and results for all aspects of the division’s functionality. Footnote 74 This suggests that it is important to implement the PM Strategy and to refine the theory of change underlying WIRA to help in targeting and optimizing changes to better meet the needs of users. To do so, the division needs to improve its logic model by clearly outlining and documenting the causal relationships between inputs, activities, outputs and outcomes. These causal relationships would articulate the needs of different clients, their points of contact with WIRA, the products they access, and how they utilize the products. This information would result in the improvement of data currently being collected and could enable the division to more efficiently and effectively target its products to users and realize its optimal capacity.

Some WIRA products are ready for Open Data Footnote 75

The division is ready for Open Data. The following tables are currently available on the Government of Canada Open Data portal:

  • Union coverage in Canada, 1997-2013;
  • Union coverage, by congress affiliation, 2013;
  • Composition of unions in Canada, by type of organization, 2013; and
  • Labour organizations with 30,000 or more covered workers, 2013.

There are an additional 19 datasets that were shared with Statistics Canada and released in February 2015, through the Canadian Socio-Economic Information Management System (CANSIM) of Statistics Canada. Footnote 76 With more WIRA products on Open Data, users are provided with opportunities to conduct their own analyses, which may reduce the number of requests for some customized tables and analyses. However, there is a chance that an increased number of complex requests from users of Open Data may result from the process.

Overall, access to WIRA products is easy

Eighty-one per cent of respondents to the User Survey who accessed WIRA products before and after 2010, reported that placing products online had a positive impact on accessibility. Footnote 77 Most users felt that accessing WIRA products and services was easy (77%). Ease of access was rated slightly higher among government users (85%) compared to non-government users (73%). A high proportion of users found WIRA products were in a convenient format (Workplace Bulletin (87%) and collective agreements (84%)). This suggests that the initiative of offering more products online met client needs. However, among 146 users who responded to the User Survey question about access, 16% reported navigation as a problem. Footnote 78 Key informants echoed this problem by saying it is essential to facilitate access through clear navigational aids and easy to understand descriptions of products, Footnote 79 suggesting that there is still work to be done to improve the usability and layout of the website to better facilitate access to WIRA products. Footnote 80

There is a need for greater awareness and promotion of WIRA products and services.

Overall, awareness of both WIRA products and research services appears to be lacking among infrequent users. Key informants reported that the division is not actively trying to increase awareness of its products. Infrequent users indicated that they would not know about new WIRA products if they did not actively search for them. It was noted that efforts were made in the past to inform users on updates to WIRA products, but these efforts were discontinued due to budgetary and time constraints. Currently, the division has no activities devoted to informing new users of its products. Footnote 81 Some key informants suggested the use of a newsletter or e-letter to inform users about new products, while surveyed users suggested using the current Workplace Bulletin to communicate the information about WIRA products. These suggestions could help to inform regular users but not new ones. In fact, the User Survey revealed that one-third (34%) of users became aware by conducting self-directed research or web searches that led them to the division’s website. Others became aware because of their job function or they were directly informed by their employer or co-worker. Footnote 82 This suggests that any strategy aimed at increasing awareness and reach to its users’ needs to be multi-faceted, taking in account self-directed as well as assisted modes. A strategy that relies on the division’s website alone or only a newsletter or a bulletin could leave a large number of potential users unaware of WIRAproducts.

For research products, several key informants noted that the new the division’s research plan consultation process and the use of SharePoint as a platform for making users aware of the available reports and analyses are both positive steps forward. These initiatives have increased the awareness of the division’s research capabilities. The consultation process has also allowed the division’s researchers and analysts to get a better understanding of the research needs of the end users of their reports. Footnote 83 However, these steps should not be seen as sufficient to ensure increased awareness and usage of WIRA products and services by internal users.

4. Conclusion

Evaluation findings show that there is an ongoing need and demand for information on collective agreements and WIRA products and services to support collective bargaining, policy development and decision-making processes in the area of labour relations.

WIRA products are predominantly viewed as enabling other branches of the Labour Program, as well as other federal departments, agencies and Crown corporations in their reporting responsibilities. In addition, WIRA support Canada in fulfilling its international reporting responsibilities by providing reports to the International Labour Organization on an annual basis. Given their national focus, WIRA products and services are unique and do not duplicate the activities of any other organizations in Canada.

WIRA products and services are reported by a majority of users and particularly by frequent users as reliable, accurate, valid, comprehensive, timely and useful in supporting collective agreement negotiations and settling labour disputes. However, the comprehensiveness of WIRA products can be further enhanced by increasing the scope of collection of information on small unions (representing under 100 employees) which is not the current focus of WIRA and conducting customized analyses by regions, occupations, industry and unions.

Changes undertaken to WIRA’s operations since 2010, including offering more WIRA products online and on Open Data have made their products available to a larger audience. The implementation of the recently developed PM Strategy is expected to collect performance information, which will contribute to further enhancements and improvements to WIRA operations and to matching WIRA products and services with their users’ needs. Having more WIRA products on Open Data will increase the usage of those products and reduce requests for some specific analyses by a variety of former and new users. Overall, the division is providing a unique service to selected Canadian and international organizations in the area of labour relations.

5. Recommendations

For WIRA to meet the needs of new, potential and regular users, it is recommended that:

1. The Workplace Information and Research Division refines its engagement strategy to increase its client base and enhance awareness of WIRA products and services.

Regular communication with users to maintain a good client base and seeking their feedback is essential to increasing uptake, usage and quality of WIRA products and services, and facilitates matching of WIRA products with users’ needs. Communication in the form of newsletters or e-letters can be considered as important media to keep most users informed about new and existing products and ways to access those products. However, multiple communication strategies would be more effective. For research planning, the division may consider building on the consultation process undertaken in spring 2014 to ensure that all users of research products, including internal analysts, are properly consulted and engaged in the research planning process. Making other labour organizations aware of WIRA products and services and exploring the possibility for expanding the current analysis threshold would help to increase the division client base.

2. The Workplace Information and Research Division enhances the online user experience by improving the navigability and the search capabilities of online databases and the division’s website.

The division should consider conducting a usability assessment of the division’s website and particularly Negotech. The issues around the navigation of the division’s website, how to use the website and where to find the needed information, are some areas for improvement. The division could coordinate with ESDC’s Innovation, Information and Technology Branch to discover appropriate ways to improve accessibility and usage of WIRA products.

3. The Workplace Information and Research Division explores ways to improve the speed of processing users’ requests to ensure that information is available in time to be useful for its internal and external users.

Timeliness is important for any information to be useful. The process and time to respond to requests for WIRA products and services need to be monitored to ensure timeliness is respected and adhered to. Efforts are needed to improve how quickly the division responds to users’ requests. It is also important that people who make special requests for WIRA products and services are informed of the turnaround times to process their requests.

4. The Workplace Information and Research Division considers implementing its new Performance Measurement Strategy to collect appropriate data to better understand and serve the needs of users.

Clearly outlining and documenting the causal relationships between inputs, activities, outputs and outcomes could offer insight into how and why users access WIRA products and services. This information could enable WIRA to more efficiently and effectively target its products and services to users and continue to effect changes that are appropriate. The PM Strategy and the division’s Theory of Change should be validated on a regular basis to ensure they continue to reflect the actual context of the division.

List of abbreviations and terms

ESDC:

Employment and Social Development Canada.

User:

Individual who accesses and uses Workplace Information and Research Activities (WIRA) products and services.

Frequent User:

Key informant who regularly uses WIRA products and services or respondent to the User Survey who used any WIRA product or service 10 or more times 12 months prior to the survey.

Infrequent or Less Frequent User:

Key informant who has limited awareness and/or usage of WIRA products and services, or respondent to the User Survey who used any WIRA product or service fewer than 10 times 12 months prior to the survey.

Known User:

Respondent to the User Survey who accesses and uses WIRA products and services and for whom the Workplace Information and Research Division had contact information.

Unknown User:

User from a union which the Workplace Information and Research Division does not track (due to low membership*), and any other user who uses and accesses WIRA products online and for whom the division has no identifying information.

Potential User:

Individual who may benefit from using WIRA products and services and who does not use them.

RDD:

Research and Data Development (merged with WID to form the Workplace Information and Research Division).

WID:

Workplace Information Division (merged with RDD to form the Workplace Information and Research Division.

WIRA:

Workplace Information and Research Activities.

The division:

Workplace Information and Research Division (new division resulting from the merger of RDD and WID). The division manages WIRA operations.

ICAMS:

Internet Client Access Management System.

TBS:

Treasury Board Secretariat.

* The division does not monitor unions with a membership below 100 in the federal jurisdiction, and unions with membership below 500 in the provincial jurisdictions.

List of figures

Figure 1: Respondents/Users by Job Description

Figure 2: Methods Used to Request WIRA Services/Products, 2008 to 2013

Figure 3: Referrals and Call-backs Trends, 2008 to 2013

Appendix A: List of key Workplace Information and Research Activities (WIRA) products

A. Published products related to collective agreements Footnote 84

Workplace bulletin

The Workplace bulletin, published monthly, is the major WIRA publication and provides a summary of:

  • Key negotiation activities;
  • Settlements reached;
  • Settlement outcomes;
  • Major work stoppages;
  • Agreement duration;
  • Working conditions of interest; and
  • A literature review.

Overview of collective bargaining in Canada

This publication provides a summary of major issues, trends and developments in collective bargaining that occurred in the previous year. It discusses:

  • Global and domestic trends impacting collective bargaining;
  • Wage adjustments;
  • Key changes negotiated in working conditions;
  • Work stoppages;
  • Union coverage; and
  • Future negotiations.

Union Coverage in Canada

Annual Union Coverage in Canada provides details on numbers of workers covered by collective agreements by type of labour organization, union size and affiliation. The publication also provides information on changes related to mergers and regulations that have impact on unions and their employees.

Data tables related to collective agreements

These data tables include, but are not limited to:

  • Ratified settlements;
  • Wages settlements;
  • Work stoppages;
  • Key negotiations; and
  • Collective bargaining calendar.

Analytical papers on issues related to industrial relations

Three papers are published online per year (started in FY 2014-2015)

B. Databases

Negotech

Negotech is an online database maintained by the division which provides full texts and summaries of collective agreements dating back to 1976. This database is searchable by official language, company name, union name, agreement or renewal number, settlement effective or expiry date, province or territory, industry, company size and union affiliation.

Directory of Labour Organizations in Canada

This database, which is updated every year through an annual survey, provides information on each union, membership and affiliation.

C. Research products

Wage projections

This annual report projects wage adjustments for the next year. The wage projections are determined for Canada overall (unionized sector), as well as the public sector and private sector for all jurisdictions and all industries.

Economic impact studies

These studies are the most common research reports prepared by the division. They assess the impact that worker actions, employer decisions and government legislation may have on the productivity and economic interests of businesses, industry and Canada as a whole.

Appendix B: Methodologies, strengths and limitations

Methodologies

The evaluation of the Workplace Information and Research Activities (WIRA) used five different data collection methods (lines of evidence) to answer the evaluation questions. The data collection methods included:

  • a document review;
  • administrative data analysis;
  • key informant interviews with Labour Program representatives;
  • a survey of users of products/ services from WIRA; and
  • a survey of potential users.

Administrative data analysis

The division administrative data analysis was conducted in an effort to determine what and how products and services are delivered, their distribution, the characteristics of the client population and the division’s performance. The administrative data included ICAMS, which is the Internal Client Access Management System. It records information on clients who are making direct contact with WIRA, either by e-mail or the 1-800 line. It contains client contact information, their organization, the nature of their request, the time to fulfill the request. Reports can be generated by a variety of criteria, e.g. client type, nature of product requested. It should be noted that ICAMS does not capture those internal users who make specific requests through methods other than e-mails and telephone calls for research and analyses.

Document review

The document review involved an examination of internal (division) and external documents. The division documents were reviewed for detailed information on division activities, costs, and performance. External documents were reviewed to provide an understanding of the context of collective agreement negotiations and industrial relations in Canadian workplaces within the federal and provincial/territorial jurisdictions. The document review also served to collect information on alternative products and services available to federal and private sector labour-management parties to help them with their collective agreement negotiations.

Strength and limitations of administrative data analysis and document review

Strengths:

  • A large number of detailed documents were available for the evaluation.

Limitations:

  • The administrative data is difficult to compare with the survey data because it is based on information from users who made requests for information to the division by e-mail, telephone or in-person. One person may have made multiple requests leading to duplicate records.
  • The administrative data collection process does not ask participants for their affiliation, while the Known User Survey did. Caution should be exercised when making comparisons between numbers from both sources in terms of user categories, product ratings or popularity.

Key informant interviews

Fourteen key informant interviews with program officials from ESDC’s Labour Program Branch were conducted to gather their opinions on the context and relevance of WIRA products and services and the extent to which WIRA products are used internally to support policy development as well as program and policy reviews. Among the 14 key informants, 5 were from the division, with the remaining participants from other Labour Program groups.

Strength and limitations of key informant interviews

Strengths:

  • The 14 key informants who participated in the evaluation were senior level management from within the division and other relevant areas within the Labour Program. Their experience, length of service and different perspectives allowed for a rich understanding of WIRA.

Limitations:

  • An important caveat to note is that those who participated in the evaluation as key informants may have a vested interest in the WIRA or partial knowledge of the division’s process and products.
  • It should be noted that for infrequent users, the reasons for not using WIRA products vary depending on the persons interviewed. Reasons such the lack of awareness, the low frequency of the need for WIRA products or the products not meeting the need of the user are all possible. It is then, not easy to separate their views about WIRA products from their respective circumstances.
  • While the report makes the distinction between WIRA products and its research services, it was not clear from the key informant interviews that users make that distinction. The division is analogous to a grocery store that has pre-packaged products, but also has a deli that provides product to their customers’ specifications. Customers come for one product and seamlessly purchase the other – both are food. The division offers information, some pre-packaged and some made to meet client-specific needs. During the interviews, respondents were asked about the accuracy of WIRA products and services. In this report the products and services have been separated out, with a focus on products, but it is likely that users’ experience with one would have a profound impact on their assessment of the other. Caution should be exercised in overly compartmentalizing the respondents’ viewpoints.

Survey of users

The survey of users of WIRA products and services was conducted to determine the extent to which each product or service is being used to inform collective bargaining and negotiation processes, for what other purposes they are used, how they meet the users’ needs and to what extent these products and services help the collective negotiation process. Surveyed users included:

  • Users of WIRA who requested information from the division by e-mail or telephone during the last five years (2008-2013);
  • Federal government users namely TBS and the Bank of Canada who receive monthly wage settlement statistics and wage projections;
  • Representatives of labour organizations holding collective agreements or in bargaining processes;
  • Others who were identified by the previous three categories of survey respondents to be users of WIRA but for whom the division does not have contact information. Unfortunately, this process did not produce enough new contacts to be helpful.

Strength and limitations of the Users Surveys

Strengths:

  • The participation rate of the Known User Survey was 22% from a total sample of 992. This provided a sufficiently large enough response base to allow for meaningful analysis.

Limitations:

  • The key limitation was the lack of information on the number and the characteristics of users who access WIRA products through the Internet only.
  • There is also a potential selection bias, as the public servants who seem to access WIRA products more frequently than others, may not have responded to the survey with the same motivation as external users.
  • The User Report included both Known and Unknown Users. Obtaining the participation of Unknown Users was a challenge; only 12 participated. These users were amalgamated with the Known Users but, as previously mentioned, were drawn from a population of unions and labour organizations.

Survey of potential users

The survey of potential users was conducted to assess awareness of WIRA products and services among individuals and organizations that are viewed as potential targets of WIRA products and services, but do not use them. In addition, the survey of potential users was also intended to examine why WIRA products and services were not being used and to explore the existence of other alternative sources of information that may be available to inform collective negotiation processes for those users.

Strength and Limitations of the survey of potential users

Strengths:

  • The participation rate of the Potential User Survey was 21% from a total sample of 513. This provided a sufficiently large enough response base to allow for meaningful analysis.

Limitations:

  • Respondents to this survey included those who were certain they had never used WIRA products or services, and those who were unsure. The dataset likely included respondents who were previous users, who stopped using WIRA products and services. The concept of a Potential User should not be limited to those who have never used the products or services before. Lapsed users may be converted to active users given the right circumstances, such as communication initiatives that correct mistaken impressions of the division’s capabilities.

Appendix C: Matrix evaluation questions and lines of evidence

Evaluation questions

  1. To what extent are WIRA products (information, analyses and research reports) accurate, reliable and relevant to types of users (unions, employers, researchers, Labour Program managers, etc.)?

    Lines of evidence used to respond to the evaluation question: User Survey, Key Informant Interviews, Potential User Survey

  2. To what extent are WIRA aligned with ESDC and the government-wide priorities and jurisdiction?

    Lines of evidence used to respond to the evaluation question: Administrative Data Analysis and Document Review, Key Informant Interviews

  3. To what extent are the WIRA duplicating or complementing or enabling other ESDC programs, initiatives or functions?

    Lines of evidence used to respond to the evaluation question: Administrative Data Analysis and Document Review, Key Informant Interviews, User Survey

  4. To what extent are WIRA products accessible to different types of users (unions, employers, researchers, labour program managers, etc.) in a timely manner?

    Lines of evidence used to respond to the evaluation question: Administrative Data Analysis and Document Review, Key Informant Interviews, User Survey, Potential User Survey.

  5. What are the impacts of WIRA products (information, analyses and research reports) on the dialogue and negotiations related to collective bargaining and on policy development in the Labour Program Branch?

    Lines of evidence used to respond to the evaluation question: Key Informant Interviews, User Survey, Potential User Survey

  6. To what extent has the business process redesign improved the efficiency of the core operational processes, enhanced services to users, and prepared WIRA for open data?

    Lines of evidence used to respond to the evaluation question: Administrative Data Analysis and Document Review, Key Informant Interviews, User Survey

Page details

Date modified: