Evaluation Report - Second Language Testing Program of the Public Service Commission

Table of contents

Executive summary

The Public Service Commission's (PSC) did an evaluation of its Second Language Testing (SLT) Program. The objective of the evaluation is to assess both the relevance and performance of SLT activities.

Under the delegated staffing system set out in the Public Service Employment Act (PSEA), the PSC requires its PSEA-delegated organizations to assess candidates' second official language skills using the PSC's Second Language Evaluation (SLE) tests, which are:

  • Test of Reading Comprehension;
  • Test of Written Expression; and
  • Test of Oral Proficiency (TOP).

SLE tests are not only used by PSEA-delegated organizations but also by non-PSEA organizations that have their own appointment authorities. Non-PSEA organizations include separate employers such as the Canada Revenue Agency, Parks Canada, the Canadian Armed Forces and the Royal Canadian Mounted Police (members).

The current SLE tests have been developed by the PSC's Personnel Psychology Centre (PPC), which is responsible for the development, monitoring and continual improvement of the three tests. The PPC oversees their administration and also trains and certifies test administrators. The PSC's Regional Offices, upon request and on a cost recovery basis, administer the tests of Reading Comprehension and Written Expression on behalf of organizations. The PSC's Montreal Regional Office also shares the responsibility of administering the TOP.

Why is this important?

There are close to 70 000 SLE tests administered annually in the federal public service. The SLE tests allow hiring organizations to evaluate the second language skills of candidates in bilingual position appointment processes. As such, these tests support the federal government's commitment to official languages within its federal institutions in terms of language of work and service to the public.

Main findings

The importance of Canada's official languages for the public service is demonstrated in legislation and government-wide priorities. The second language testing provided by the PSC is aligned with these priorities and with the PSC's strategic outcome, as it supports merit in appointments.

Second language testing is responding to a continuous need, not only for staffing purposes, but also for the professional development of public service employees. This is true for PSEA and non-PSEA organizations alike.

Roles and responsibilities as they relate to official language testing are shared between the Treasury Board, the PSC and hiring organizations. These partners are well aware of their respective roles and responsibilities.

The majority (above 80%) of corporate users surveyed expressed satisfaction with the SLE testing services in general and over 90% of test takers surveyed were satisfied with their overall testing experience. Still, improvements are needed in some areas. For example, the Program has not been meeting its scheduling standards for the TOP over the last two years and dissatisfied corporate users reported having difficulty in reaching the appropriate PSC person when additional help was needed.

While the information provided online to help test takers prepare for their tests is found useful by some, many of them are not aware of it.

There is a gap between the performance feedback information desired by test takers of the TOP and the information provided in the standard performance feedback they received.

Although not specific to SLT activities, the evaluation found that there is a portion of test takers with disabilities who decided not to request accommodation (3% of tests takers surveyed). The main reason cited for this were fear of negative consequences, finding the process of requesting accommodation too cumbersome or being unaware of their right to accommodation.

There are adequate mechanisms to ensure continued improvement of test content and test security for all three SLE tests. Quality control monitoring for the TOP is also adequate.

Efficiency gains were made by making online testing mandatory for the reading comprehension and written expression tests. However, the Program's heavy reliance on technology makes it vulnerable to information technology system interruptions.

Other approaches to assess second official languages exist, but caution needs to be exercised as not all alternatives provide the assurance that official languages standards are applied uniformly across hiring organizations.

Recommendations

Based on the above findings, the evaluation recommends the following:

Recommendation 1: The PPC/PSC should review its client service entry points to identify options for their optimization.

Recommendation 2: The PPC/PSC should reassess its current resourcing model for second language assessors, including the number of full-time equivalents needed for SLE TOP administration, in order to be more responsive to client demand.

Recommendation 3: The PSC should review its outreach efforts and the way information on language requirements of positions and SLE tests is presented online in order to better reach its targeted audience.

Recommendation 4: The PSC should review the standard feedback it provides to TOP test takers to identify improvements to better meet the needs of test takers, while balancing the need for test security. The PSC should also study the feasibility of offering a less resource-intensive version of its detailed feedback at a lower cost.

Recommendation 5: The PSC should seek additional avenues to promote candidates' rights to assessment accommodation and to a fair, barrier-free evaluation of their abilities. It should also review its accommodation processes to identify any potential barriers to accommodation and measures to eliminate them.

1. Purpose of the evaluation

The objective of the evaluation is to assess both the relevance and performance of the Second Language Testing (SLT) Program through the following eight questions:

  • To what extent does the Program continue to address a demonstrable need?
  • To what extent is the Program aligned with government-wide priorities and the Public Service Commission's (PSC) strategic outcome?
  • Are roles and responsibilities clear and well communicated?
  • Are the design and administration of the Program meeting the needs of corporate users?
  • Are the design and administration of the Program responsive to the needs of test takers?
  • To what extent are the Second Language Evaluation (SLE) test processes sound and reliable (test development, maintenance, training materials, quality control, test security, etc.)?
  • Is the Program managed in an efficient and cost-effective manner (including the optimal use of technology)?
  • Are there other approaches to assess candidates' second official language?

The evaluation of the PSC's SLT Program was initiated by the President as part of the PSC's Three-Year Risk-Based Audit and Evaluation Plan for 2015-2018.

Corporate users of the Program include organizations that are governed under the Public Service Employment Act (PSEA) for their appointments and organizations that have their own appointment authorities; this includes separate employers such as the Canada Revenue Agency, Parks Canada, the Canadian Armed Forces and the Royal Canadian Mounted Police (members). This latter group of organization will be referred in this report as non-PSEA organizations.

2. Description of the Second Language Testing Program

Under the delegated staffing system set out in the Public Service Employment Act (PSEA), the Public Service Commission (PSC) requires, through its Appointment Policy, that deputy heads use Second Language Evaluation (SLE) tests approved by the PSC to assess the proficiency in the second official language of candidates in appointment processes.

Second official language qualifications are established by the employer, Treasury Board (TB), and are part of the Qualification Standards in Relation to Official Languages. The qualifications apply to core federal public service positions that require the use of both official languages. The Standards define three levels of proficiency (A, B and C – C being the highest level) for the following three language skills:

  • Written Comprehension in the Second Official Language;
  • Written Expression in the Second Official Language; and
  • Oral Proficiency in the Second Official Language.

At the present time, the only PSC approved instruments to assess second official language qualifications are the following SLE tests:

SLE - tests
*Unless accommodation is needed or for other exceptional circumstances, upon Personnel Psychology Centre (PPC) approval.
SLE – Test of Reading Comprehension
It assesses the ability to understand texts written in a person's second official language. Consists of multiple-choice questions and is administered online* through the PSC Online Testing Facility (OLTF). The test is computer-generated (CGT) and uses a delivery approach that automatically builds a different test for each examinee, according to pre-determined content specifications For PSEA-delegated organizations:
  • Test is free of charge
  • Can be administered by the PSC on behalf of delegates under cost recovery
For non-PSEA organizations:
  • Test and administration services are offered under cost recovery
SLE – Test of Written Expression
It assesses the knowledge of second language grammar, vocabulary and other aspects of written expression that are necessary to perform writing tasks dealing with work-related situations. Consists of multiple-choice questions and is administered online* through the OLTF. The test is also a CGT that automatically builds a different test for each examinee, according to pre-determined content specifications. For PSEA-delegated organizations:
  • Test is free of charge
  • Can be administered by the PSC on behalf of delegates under cost recovery
For non-PSEA organizations:
  • Test and administration services are offered under cost recovery
SLE – Test of Oral Proficiency
It assesses the ability to speak and understand one's second official language. It includes four parts and is administered by a PSC-certified assessor by telephone or face-to-face. Questions asked are also generated by a computer and a different test for each examinee is built according to pre-determined content specifications. This test also has an adaptive component. As such, based on the level of proficiency demonstrated by the test taker, the assessor decides whether to administer only two, three or all four parts of the test For PSEA-delegated organizations:
  • Use of the test is free of charge for PSEA organizations
  • They are only charged for late cancellations and no shows
For non-PSEA organizations:
  • Test and administration services are offered under cost recovery
Organizations can administer the tests themselves when they have a PSC-certified second language assessor on staff. At the time this evaluation was conducted, no organization was using this option

All three SLE tests have a 30-day waiting period before a person can be retested. Test results have a validity period of five years, but may remain valid indefinitely for persons who remain in the same position, provided that the linguistic profile of the position is not raised above the person's skill level while in the position. To ensure employees have maintained the required level of language proficiency, a manager may request that an employee be tested prior to or after the five year lapse period. An exemption from further second language testing is granted to persons who obtain a high enough score that they need not be tested again.

2.1 Public Service Commission's second language testing activities

There are close to 70 000 SLE tests administered annually in the federal public service. These tests allow organizations to evaluate the second language skills of candidates of bilingual positions. As such, they support the federal government's commitment to official languages within its federal institutions in terms of language of work and service to the public.

Under the PSC 2015-2016 Program Alignment Architecture, the SLT Program is within the Staffing and Assessment Services Branch, more specifically within the Personnel Psychology Centre (PPC) and the Central Programs and Regional Offices (CPRO) Directorate. The SLT budget is about $4 million a year. The revenues generated by SLT services were about $1.5 million a year over the last four years, accounting for about 20% of all PSC revenues. The Program can be divided into three groups of activities:

Research and development

The research and development (R&D) activities related to SLE are conducted by the R&D Division of the PPC. This division develops, monitors and continually updates all three SLE tests. Tests are developed according to professional standards and principles of Universal Test Design. The R&D Division oversees the development of alternative test formats for accommodating persons with disabilities. It is also responsible for defending the tests when complaints are made to the Public Service Labour Relations and Employment Board, or to other tribunals or courts.

Service delivery

The PPC's Consultation and Test Services Division (CTS) and the Assessment and Counselling Services (ACS) Division, as well as CPRO, deliver test administration services (this also includes scheduling, corrections, dissemination of results and feedback). Organizations can administer Tests of Written Expression or Reading Comprehension themselves, or they can access PSC test administration services. Within the PSC, CTS is in charge of all online testing services as well as determining assessment accommodation for all SLE tests. Additionally, it certifies online test administrators and ensures follow-up on corrective measures when incidents occur during testing sessions. ACS is also in charge of all Tests of Oral Proficiency (TOP) administered in the National Capital Region, while CPRO is in charge of administering tests in PSC regional offices.

Monitoring and quality control

The Quality Control Unit is part of the PPC's ACS Division and is responsible for monitoring and quality control activities for the SLE TOP. They ensure continual training and certification of all PSC-certified TOP assessors.

Table 1 below presents the estimated full-time equivalents (FTE) for each group of activities.

Table 1: Estimated second language testing full-time equivalents
Activities Estimated FTEs
Source: Personnel Psychology Centre
PPC Activities:
     Research and development 9.5
     Service Delivery 31.3
     Monitoring and Quality Control (for SLE TOP only) 6.7
CPRO Activities :
     Service Delivery 23.1
Total 70.6

3. Evaluation scope and design

3.1 Objectives and scope

The evaluation examined the relevance and performance of the Second Language Testing (SLT) Program conducted by the Public Service Commission (PSC), in accordance with the Treasury Board (TB) Policy on Evaluation and Directive on the Evaluation Function. The evaluation did not have as an objective to look at the validity of the Second Language Evaluation (SLE) tests and their ability to determine second language proficiency.

The evaluation was conducted between January and December 2015. This is the first evaluation of the SLT Program of the PSC. The Program's data was examined over a five-year period from 2010-2011 to 2014-2015.

3.2 Methodology

The evaluation used a mix of quantitative and qualitative methods, incorporating multiple sources of evidence. More specifically, the following methods were used (associated limitations can be found in Appendix 1):

Table
Methods Used Description
Document review
  • Background and mandate of the Program
  • Operational documents (such as test manuals, instructions, etc.)
  • Strategic organizational and government documents
Literature review The following themes were explored:
  • Models for test delivery and administration of large-scale standardized tests
  • Issues in the delivery and administration of tests
  • Examples of second language and large-scale testing activities outside the public service
17 Interviews
  • Test takers (5)
  • Human resources (HR) personnel (5)
  • PSC management (7)
Data analysis
  • All PSC systems data for SLT activities for years 2010-2011 to 2014-2015
  • Staffing activities by language requirements for years 2010-2011 to 2014-2015
  • SLT activities financial data
Surveys
  • HR personnel – 737 respondents (response rate of 43%)
  • TOP test takers – exit survey, 412 respondents
  • Written Expression and Reading Comprehension test takers – exit survey, 1 515 respondents
Respondents were asked to rate their answers on a five-point Likert scale from positive to negative. There was a neutral middle to the scale. In the reporting of the survey results, positive and negative answers are collapsed

4. Findings

4.1 Relevance: Continued need for the Program

To what extent does the Public Service Commission's Second Language Testing Program continue to address a demonstrable need?

Second language testing is responding to a continuous need not only for staffing purposes but also for the professional development of public service employees. This is true for Public Service Employment Act (PSEA) and non-PSEA organizations alike.

We examined the Program administrative data to see the Program usage trends over the recent past. We also looked at the number of bilingual positions in the federal public service.

Usage data shows a continuous need for the Program, not only for staffing purposes but also for employee development

As shown in Figure 1, the demand for Second Language Evaluation (SLE) tests for staffing purposes decreased following the workforce adjustment of 2011-2013. Still, there was a 14% increase in demand in 2014-2015. Preliminary data for the first two quarters of 2015-2016, which are not included in the figure, were also on the rise.

Figure 1 also shows that the demand for SLE tests for category “record purposes, re-identification and other” has remained quite stable over time. People being tested in this category include individuals who have expired SLE test results and who want, for mobility or other reasons, to maintain valid SLE results. It also includes individuals whose language profile has changed and need to be tested in light of the new profile, as well as individuals being tested for bilingualism bonus eligibility, as per the Bilingualism Bonus Directive of the National Joint Council.

Figure 1: Number of Second Language Evaluation tests administered by reasons for testing and year

Figure 1
Long Descriptions for Figure 1
Figure 1: Number of Second Language Evaluation tests administered by reasons for testing and year
Reason for testing 2010-2011 2011-2012 2012-2013 2013-2014 2014-2015
Staffing Imperative 43 727 28 670 21 937 21 694 29 823
Record Purposes, Re-Identification and Other 28 322 31 002 29 438 30 430 31 651
Training 8 692 7 978 6 185 5 807 5 437
Staffing Non-imperative 3 801 3 439 2 424 3 309 2 801

Source: Public Service Commission Test Scoring and Results Reporting System

When we looked more specifically at the data regarding the SLE Test of Oral Proficiency (TOP), we found that the proportion of test takers who have taken the TOP for purposes other than staffing has constantly accounted for more than 50% of all tests administered over the last five years.

Data also reveals that the usage made of the second language testing Program by non-Public Service Employment Act (PSEA) organizations also remained stable across the last five years, accounting for about 40% of all tests administered by year.

Number of appointments to bilingual positions in the public service remained stable

When we looked at staffing activities under the PSEA by language requirements of position over the last five years, we noted that the proportion of bilingual positions remained stable, between 45 and 46%.

Table 2: Proportion of staffing activities conducted under the Public Service Employment Act to bilingual positions, by year

Table 2: Proportion of staffing activities conducted under the Public Service Employment Act to bilingual positions, by year
Source: Public Service Commission hiring and staffing activity files
2010-2011 2011-2012 2012-2013 2013-2014 2014-2015
46% 44% 44% 43% 45%

4.2 Relevance: Alignment with government priorities and strategic outcome

To what extent is the Program aligned with government-wide priorities and the Public Service Commission's strategic outcome?

The importance of Canada's official languages is demonstrated in legislation and key strategic government documents. The Second Language Testing Program is also aligned with the Public Service Commission's strategic outcome, as it supports merit by thoroughly assessing the second official language proficiency of public servants.

We examined legislation and government-wide strategic documents to see if official languages (OL) obligations were a priority of the federal government. We also looked at whether there was alignment between the Second Language Testing (SLT) Program and the strategic outcome of the Public Service Commission (PSC).

Official languages and bilingualism of the public service is a priority for the federal government

Through the Official Languages Act (OLA), which was given quasi-constitutional status by the courts, the Government is to ensure respect for English and French as the official languages of Canada and ensure equal status and equal rights and privileges as to their use in all federal institutions. To support the Government commitment to OL, both the House of Commons and the Senate have standing committees to review and study questions, policies and programs related to OL.

In the Speech from the Throne to open the First Session of the Forty-second Parliament of Canada, the Government pledged to “encourage and promote the use of Canada's official languages”. More specifically, the Minister of Canadian Heritage has been mandated to work with the President of the Treasury Board to ensure that all federal services are delivered in full compliance with the OLA.

The importance of official languages is expressed in key government documents

In the Twenty Second Annual Report to the Prime Minister, the Clerk of the Privy Council expressed that the public service “[is], and must be, reflective of Canada's diversity and active in championing our official languages, including in our workplaces”.

Also, as part of the Destination 2020 initiative, the Council of the Network of Official Languages Champions spoke about the importance of leveraging bilingualism in the public service of the future, stating that “[a] world class Public Service equipped to serve Canada and Canadians now and in the future involves striving for a more bilingual Public Service…”.

Second Language Testing Program is aligned with the Public Service Commission strategic outcome

The 2015-2016 strategic outcome of the PSC is “a highly competent, non-partisan and representative public service, able to provide service in both official languages, in which appointments are based on merit and the values of fairness, access, representativeness and transparency”. The PSC ensures that its tests are accessible to all test takers through the principles of Universal Test Design and by using multiple formats.

The SLT activities are aligned with this outcome as they allow for the merit criteria of second official language proficiency to be assessed for bilingual positions. In doing so, they also contribute to a public service able to provide services to Canadians in both official languages.

4.3 Relevance: Roles and responsibilities

Are roles and responsibilities clear and well communicated?

Within the Public Service of Canada, roles and responsibilities in relation to second official languages testing are shared between the Treasury Board, the Public Service Commission and hiring organizations. Although these partners' respective roles are defined and communicated, interview data reveals that some test takers are confused regarding who is responsible for what.

Responsibilities as they relate to the SLT Program are shared amongst many federal organizations. As a result, there should be clearly defined roles and responsibilities for each of the players. We examined Program documentation, held interviews with key stakeholders and conducted surveys of the Program users to determine if roles and responsibilities are clearly defined, well-aligned and communicated.

Roles and responsibilities were aligned, defined and communicated — still, they were not always clear to test takers

A review of documentation allowed us to determine that the roles and responsibilities of the Treasury Board (TB), the PSC and hiring organizations, as they relate to SLT, are well defined. The following is a summary of the main roles and responsibilities of each group.

Table 3: Roles and responsibilities of key stakeholder groups
* Through the administration of the Bilingualism Bonus Directive of the National Joint Council
Roles and Responsibilities Treasury Board
Secretariat
Public Service
Commission
PSEA
Organizations
Non-PSEA
Organizations
Establish Official Languages Qualification Standards Treasury Board Secretariat has the responsability to establish official languages qualification standards     Non-Public Service Employment Act Organizations has the responsability to establish official languages qualification standards
Set language profile of given position     Public Service Employment Act Organizations has the responsability to set language profile of given position Non-Public Service Employment Act Organizations has the responsability to set language profile of given position
Choose assessment tool to assess second language proficiency   Public Service Commission has the responsability to choose assessment tool to assess second language proficiency   Non-Public Service Employment Act Organizations has the responsability to choose assessment tool to assess second language proficiency
Develop and maintain the SLE tests   Public Service Commission has the responsability to develop and maintain the Second language evaluation tests    
Administer SLE tests   Public Service Commission has the responsability to administer Second language evaluation tests Public Service Employment Act Organizations has the responsability to administer Second language evaluation tests Non-Public Service Employment Act Organizations has the responsability to administer Second language evaluation tests
Ensure that employees maintain their language skills Treasury Board Secretariat has the responsability to ensure that employees maintain their language skills Public Service Commission has the responsability to ensure that employees maintain their language skills through the administration of the bilingualism bonus directive of the National Joint Council Public Service Employment Act Organizations has the responsability to ensure that employees maintain their language skills Non-Public Service Employment Act Organizations has the responsability to ensure that employees maintain their language skills

These roles and responsibilities are communicated through the Treasury Board Secretariat and the PSC websites as well as the jobs.gc.ca website. Still, interview data highlights some level of confusion for some test takers who do not differentiate between the role of the PSC and that of TB. As a result, the PSC does, at times, receive complaints that do not fall under its role or responsibilities.

Although the evaluation recognized that some candidates are confused in relation to the roles and responsibilities of the PSC, TB and hiring organizations, the associated number remains small (less than 5 complaints a year). As such, the evaluation does not see a need for a recommendation specific to the subject of roles and responsibilities. However the evaluation does address, through recommendation 3 of this report, the need for the PSC to review how information on language requirements and on SLT is presented on its multiple webpages to ensure that all information made available reaches its target audience.

Roles and responsibilities are shared amongst several individuals within client organizations

The evaluation also looked at roles and responsibilities within hiring organizations, as they relate to SLT activities. Through SLE test administration manuals and other supporting documents provided to client organizations, the PSC clearly defines the roles and responsibilities of clients as they relate to SLT activities and the use of its associated information technology (IT) systems.

The survey administered to human resources (HR) personnel highlighted that, within hiring organizations, responsibilities related to SLT are also shared amongst several individuals. For example, one person will administer online tests, a second will do the scheduling for the SLE TOP, a third will act as an on-site responsible officer for TOPs administered by phone, etc.

4.4 Performance: Meeting hiring organizations' needs

Are the design and administration of the Program meeting the needs of corporate users (organizations under the Public Service Employment Act and non-Public Service Employment Act organizations)?

The majority (above 80%) of corporate users surveyed expressed satisfaction with the Second Language Evaluation testing services, the material provided and the information technology systems used; still, dissatisfied corporate users reported having difficulty in reaching the appropriate Public Service Commission person when additional help was needed.

For the Test of Oral Proficiency, meeting the scheduling standards has been a challenge over the last two years and user satisfaction on that level is low.

One of the main goals of the SLT Program is to allow hiring organizations to effectively assess the essential qualification of second language proficiency while maintaining the efficiency of the staffing process. We therefore assessed the level of satisfaction with testing activities amongst corporate users.

Corporate users were satisfied with the second language testing information technology systems used

Corporate users of the Program interact with four different IT systems, which are:

  • Test Inventory Control System (TICS) – allows corporate users to order test material;
  • SLE scheduling system (SLE-SS) – allows corporate users to submit requests to have candidates tested in their second language by the PSC (any combination of Reading Comprehension, Written Expression and Oral Proficiency);
  • Online Testing Facility (OLTF) – online platform in which electronic master tests are stored; it is used by certified organizational test administrators to administer SLE online tests; and
  • Test Result Search Tool (TRST) – allows corporate users to retrieve official SLE test results electronically for candidates who have a personal record identifier.

The SLT Program also uses the Online Oral Proficiency System to store and play digital audio recordings of questions selected for the administration of the TOP. Since there was no client organization that had a TOP assessor on their staff at the time this evaluation was conducted, there was no corporate client surveyed regarding it.

The majority of corporate users agreed that the IT systems associated with SLT activities are easy to use. More specifically, 86% of corporate users expressed satisfaction with the ease of use of the SLE-SS. That number was 81% for TICS, 87% for OLTF and 93% for the TRST.

Still, as stated in section 4.3 of this report, responsibilities of corporate users are decentralized amongst several individuals working for the client organization. This reality is also reflected in the user list for each system. As such, although there is some overlap between names that appear on the user lists of all four systems, it could be that a user of one specific system is not a user of the three other systems. This situation can create duplication in communication for the PPC who, to ensure that information it communicates to organizations reaches the right people, has to communicate it through several lists of users.

Although this creates, at times, additional work for the Program, the Program is confident that the current multi-year initiative led by the PSC's Information Technology Services Directorate to rationalize the PPC's IT applications will correct the situation. Through this initiative, PPC's IT systems will be streamlined and modernized. As a result, communication with corporate users will most likely become easier as the number of client lists will also be reduced.

Corporate users were satisfied with the quality of service received when directly contacting the Public Service Commission, but expressed having difficulties in reaching the right person

Over 81% of users expressed satisfaction with the service provided when they contacted the PSC to obtain additional information or help. Still, qualitative data collected from users who expressed dissatisfaction revealed that the two most common sources of dissatisfaction were difficulty reaching the appropriate PSC/PPC person and a perceived lack of support when testing issues arose with the OLTF platform.

The PSC's role regarding the OLTF platform is to help clients find solutions in relation to what should be done if an incident occurs during a testing session. The PSC cannot support every operating system used by its corporate clients. The PSC can only provide limited support to clients, who ultimately need to contact their own organizational IT support team for help. To this end, the Memorandum of Understanding established between the PSC and organizations stipulates that the PSC will not be responsible for system interruptions beyond its control, such as those arising from Shared Services Canada's repair or maintenance activities.

As for the reported difficulty in reaching the appropriate PSC/PPC person, the evaluation also conducted a review of all client service entry points available for SLE-related issues. We found that there are a high number of client service entry points and that it is not always clear who needs to be contacted.

For example, in relation to SLE TOP scheduling issues, there is a list of six generic email addresses associated with the icon “Contact Us”. Another example is that the “Contact Us” icon associated with the extranet page for SLE test center coordinators identifies many generic email addresses and contact numbers, including those for the political activities general information line and Priority Administration.

Recommendation 1:

The PSC/PPC should review its client service entry points to identify options for their optimization.

The Public Service Commission was not able to meet its scheduling standards for the Test of Oral Proficiency over the last two years

For the administration of the TOP, the PSC has three different scheduling standards, depending on the reason for testing, detailed in Table 4. In its Performance Management Framework, the PSC has set a target of meeting these standards 95% of the time. Although the PSC was able to attain that target prior to 2013-2014, it has not been met over the last two years and performance is in decline.

Table 4: Percentage of time the Test of Oral Proficiency scheduling standards were met
Source: PSC Second Language Oral Interaction Evaluations Schedule System
Standard 2013-2014 2014-2015
Imperative staffing for an impending appointment – 10-day scheduling standard Met 47% of time Met 36% of time
Language training – 20-day scheduling standard Met 82% of time Met 49% of time
Non-imperative staffing or other – 50-day scheduling standard Met 89% of time Met 20% of time

Human resources constraints, combined with a 19% demand increase over the last two years, and a backlog resulting from the PSC data center move, explains in part the challenges experienced by the Program. Related survey satisfaction results with the service are also low, as 46% of corporate users expressed dissatisfaction with the delay in meeting the 10-day standard, 36% for the 20-day standard, and 57% for the 50-day standard.

The TOP is administered by a team of specialists (EDS-1) that has been challenging to maintain in sufficient numbers. As a contingency measure, the PSC planned to overstaff the team by two full-time equivalents (FTE) for each language. While the PPC succeeded in hiring 10 new assessors in 2015-2016, staffing processes were not productive enough to sufficiently overstaff.

Recommendation 2:

The PSC/PPC should reassess its current resourcing model for second language assessors, including the number of FTEs needed for SLE TOP administration, in order to be more responsive to client demand.

4.5 Performance: Responsiveness to test takers' needs

Are the design and administration of the Program responsive to the needs of test takers?

Over 90% of test takers surveyed were satisfied with their testing experience. Still, data shows that some of them are not aware of the information available on the PSC website to help with test preparation.

Survey and interview data highlight a gap between the performance feedback desired by test takers of the Test of Oral Proficiency and the information provided in the standard performance feedback they received.

Data also shows that although 86% of test takers were aware of their accommodation rights, some test takers did not request accommodation because they were not aware of their rights. Data also shows that some tests takers who required accommodation chose to waive their right either for fear of negative consequences or because the accommodation process was felt to be too cumbersome.

Test takers' overall satisfaction with the SLT Program is important to meeting Program outcomes. We therefore examined the satisfaction of test takers with their overall testing experience. We also looked at the test information made available for test takers and their views on the usefulness of this material to their test preparation. In addition, we examined test takers' views on the performance feedback they received for the TOP.

Test takers were satisfied with the overall second language testing process

The survey administered to test takers shows that over 90% of them were satisfied with their testing experience for all SLE tests. More specifically, over 90% of them were satisfied with the testing conditions and the environment, while over 97% of them found the instructions provided to be clear. Specific to the TOP, over 97% of test takers found the audio clips used were easy to hear.

Qualitative data collected through the survey and the interviews also highlights that several TOP test takers appreciated the manner displayed by the second language assessor, which was reported as cordial, polite and positive.

Test takers found communication material provided for test preparation useful — however, some test takers were not aware that this material was available

Seventy-nine percent of test takers surveyed who took the tests of written expression and/or reading comprehension were aware that information was available on the PSC website to help them with their test preparation, while 21% reported not being aware. Of those who were aware that information was available, 45% learned about it through the organization that requested they be tested, while 37% learned it either through their own research or through colleagues and friends. Of those who consulted the material, 71% agreed that the information was useful to their preparation.

In relation to the test takers for the TOP, survey data indicated that although 80% of them were aware that there is some information on the PSC website to help with their preparation, they were rarely aware of all the information available nor did they consult all of it. Of those who were aware that information was available, 51% learned about it through the organization that requested they be tested, while 34% learned it either through their own research or through colleagues and friends.

For the SLE TOP, the information available online includes:

  • A general information brochure;
  • A video that walks test takers through the four parts of the test and provides them with details of what they can expect for each part; and
  • Several audio clip samples that explain what is assessed at the levels A, B and C, along with excerpts of interviews at each level.

The evaluation found that 63% of TOP test takers either did not consult or were not aware of the existence of the video. That number was 43% for the audio clips and 26% for the general information brochure.

Still, we found that of those who consulted the information, 81% agreed that the general information brochure was helpful for the exam, while 66% agreed that the video was helpful, and 83% agreed that the audio clips were helpful.

In addition, and as discussed under section 4.3 of this report, the PSC also provides candidates with general information on roles and responsibilities and on language requirements of positions.

All information available for test takers is available through multiple webpages that are not necessarily linked together, which makes it more challenging to ensure that candidates access all the information available. Three categories of information available for test takers consist of:

  • Information specific to the three SLE tests, for which the PPC is the content owner;
  • Information on the roles and responsibilities and language requirements of positions, for which CPRO is the content owner; and
  • Information on language requirements available on the jobs.gc.ca website, for which the Business Development and Systems Directorate of the PSC is the content owner.

Recommendation 3:

The PSC should review its outreach efforts and the way information on language requirements of positions and SLE tests is presented online in order to better reach its targeted audience.

A significant number of test takers expressed dissatisfaction with the test performance feedback they received for the Test of Oral Proficiency

Following their testing session, test takers of the TOP received a one page performance feedback document. This feedback is automated and written in the form of “can do” statements. The feedback highlights what the test taker was able to demonstrate during the testing session that corresponds to the level of proficiency they attained, rather than what they were not able to demonstrate in order to achieve the next level of proficiency. Some test takers reported that they were left perplexed as to what is needed for them to achieve the next level of proficiency, since their feedback sounds “good” overall.

A review of sample feedback revealed that feedback of contiguous levels of proficiency can have up to four sections out of six that are identical. These similarities between levels add to the difficulty amongst test takers to grasp why they were not able to reach the next level of proficiency. Refer to Appendix 2 to see samples of feedback for proficiency levels B and C.

Survey data reveals that of the TOP test takers surveyed who received feedback within the last two years, 33% agreed that the feedback was useful, while 46% disagreed and 21% remained neutral.

The PSC does offer more detailed feedback under cost recovery. This feedback provides candidates with a detailed picture of their most recent test performance. Feedback is given with the aid of both detailed analysis and examples taken directly from the test in question. To be admissible to receive a detailed feedback, a candidate must have taken the TOP within eight weeks of the date that the contract was signed, must be enrolled in language training and must attend the feedback session with their language training representative. This service is relatively expensive (1.28 x the per diem rate for professional services offered by SASB) and is relatively unknown (70% of HR personnel surveyed were unaware of the service). Forty-nine detailed feedback sessions were provided in 2014-2015, 34 in 2013-2014 and 26 in 2012-2013.

Recommendation 4:

The PSC should review the standard feedback it provides to TOP test takers to identify improvements to better meet the needs of test takers, while balancing the need for test security. The PSC should also study the feasibility of offering a less resource-intensive version of its detailed feedback at a lower cost.

For the tests of written expression and reading comprehension, test takers do not receive any performance feedback. Data collected through the survey and interviews did not identify a clear need from test takers to obtain additional feedback on these two tests. However, data allowed us to identify a desire from test takers to have access to more practice questions than the few available on the PSC website (when the evaluation was conducted, there were two sample questions for the Test of Written Expression and four sample questions for the Test of Reading Comprehension). Upon completion of the evaluation data collection phase in December 2015, the PSC launched two full SLE self-assessment practice tests. One is for the Test of Written Expression and one for the Test of Reading Comprehension. Both self-assessments have 40 questions.

Efforts have been made by the Program to develop inclusive tests — still, the need for individual accommodation remains

The PSC has the policy authority for assessment, which includes the duty to accommodate individuals. This duty to accommodate provides an equal opportunity for all candidates to demonstrate that they meet the qualifications as measured by the assessment. Interview data and documents reviewed shows that the PPC has invested significant efforts in recent years to introduce features of Universal Test Design into its SLE instruments. These design elements are intended to reduce the need for assessment accommodation by making the tests accessible to as broad a population as possible. For example, changes made to item length, format and content can reduce barriers for individuals with attention, memory or visual deficits. While these efforts are a step in the right direction, a need for individual accommodation remains. As such, information provided by the Program to test takers, through the PSC website, informs test takers of their accommodation rights.

Some test takers are not requesting accommodation although they would be entitled to it

Test taker survey data shows that 86% of respondents were aware of their accommodation rights prior to the testing session. While 95% of all test takers surveyed did not request accommodation because they did not need it, 2% (39) of them declared having requested accommodation and 3% (44) declared that, while they would have needed accommodation, they decided not to request it. The three main reasons cited for not requesting accommodation were fear of negative consequences, the accommodation process was felt to be too cumbersome and being unaware of accommodation rights.

Of the test takers surveyed who received accommodation, 82% of them expressed satisfaction with the accommodation process overall, while 6% remain neutral and 12% expressed dissatisfaction. Although test takers surveyed who expressed dissatisfaction with the process were asked to provide additional comments, few did. Of those who provided comments, the most cited reason was that the process was too cumbersome. These results regarding the accommodation process were echoed in the survey administered to HR personnel, where 77% of users express satisfaction with the process, while 17% remained neutral and 6% expressed dissatisfaction. The reason most cited behind the dissatisfaction of HR personnel related to the delays associated with the process.

This finding is not specific to the SLT Program and the sample supporting it is small. Still, considering PSC obligations under the Employment Equity Act, to identify and eliminate employment barriers, we think that a recommendation to review the accommodation process is required.

Recommendation 5:

The PSC should seek additional avenues to promote candidates' rights to assessment accommodation and to a fair, barrier-free evaluation of their abilities. It should also review its accommodation processes to identify any potential barriers to accommodation and measures to eliminate them.

4.6 Performance: Sound and reliable processes

To what extent are the Second Language Evaluation test processes sound and reliable (test development, maintenance, training materials, quality control, test security, etc.)?

There are adequate mechanisms to ensure continued improvement of test content and test security for all three Second Language Evaluation tests. Quality control monitoring for the Test of Oral Proficiency is also adequate.

The evaluation examined the existence and adequacy of mechanisms to maintain and update test content and to ensure test security. We looked at the quality of the testing material and the training provided. We also reviewed the quality control process for the TOP.

Mechanisms were in place to maintain and update test content and ensure test security

As required by the document “Testing in the Public Service of Canada”, the validity and reliability of the SLE tests are studied and well documented. The PSC monitors the performance of the SLE tests periodically and ensures that the tests are fair to employment equity groups. Having computer-generated tests (CGT) for the reading comprehension and written expression tests, combined with their mandatory online administration, allow the PSC to pilot new test items on a continual basis. The PPC has a multidisciplinary unit (comprised of psychologists and education specialists) dedicated to SLE research and development. This unit applies professional standards in the development of the SLE tests.

CGT and online administration also increased test security. The PSC now has a greater number of test versions compared to paper-and-pencil tests and can electronically track test access.

Corporate users were satisfied with the quality of testing material and the administrator training provided

For the written expression and reading comprehension tests, over 84% of corporate users surveyed found the test administration manuals and associated instructions were clear. As for the TOP, 92% of corporate users expressed satisfaction with the clarity of the instructions provided to the on-site officers who monitor the administration of the TOP done over the phone at the hiring organization's location.

The evaluation also found that 86% of corporate users expressed satisfaction with the certification training they received to become certified online test administrators. This training allows them to administer the Test of Written Expression and the Test of Reading Comprehension online through the OLFT platform.

The quality control associated with the Test of Oral Proficiency was adequate

To ensure inter-rater reliability for the TOP, monitoring of second language assessors is done. Inter-rater reliability is the degree of agreement amongst the assessors. As one element of the quality control program, about 9% of all TOPs administered are submitted to different degrees of quality control — for example, blind rating of recording, continual monitoring of assessors, rescore request, etc.

Administrative data also shows that the number of complaints for the TOP are low. They represent about 0.2% of all TOPs administered. The number of requests for TOP rescoring is also low: Less than 5% of all TOPs administered.

Survey data reveals that 61% of TOP test takers were aware of the possibility to ask for a rescore at the time of their testing. This number appears positive as it is a common practice for HR personnel to formally inform candidates of their recourse rights only once results are known. For example, candidates are usually made aware of the possibility for informal discussion when they receive a notice informing them that they were eliminated from the process. This is also true of recourse rights in front of the Public Service Labour Relations and Employment Board, of which candidates are formally made aware through the notification process shortly prior to when an appointment is made.

Still, the PSC is not relying on the HR community to inform candidates of their recourse rights in relation to SLE testing. As such, information made available online to test takers regarding test preparation also informs them of their recourse rights through a page of questions and answers. The evaluation team also believes that if recommendation 3 of the present report is fully implemented, the percentage of test takers aware of the possibility of rescore at the time of their testing will most likely increase.

4.7 Performance: Efficiency and economy

Is the Program managed in an efficient and cost-effective manner (including the optimal use of technology)?

Efficiency gains were made by making online testing mandatory for the reading comprehension and written expression tests. However, the Program's heavy reliance on technology makes it vulnerable to information technology service interruptions.

Effective assessment tools and processes contribute to the quality of hires in the public service. IT used in testing can improve these processes, but may also hinder them, depending in part on their adaptability. The evaluation examined to what extent the use of technology enables quick, affordable and reliable testing.

Important efficiency gains have been made on many fronts

Over 90% of SLE tests are now administered online. The move from paper-and-pencil tests to online testing allowed for a reduction in service turnaround times, from 15 business days to 24 hours.

The use of computer-generated tests (CGT) enables the PSC to respond rapidly to possible test security incidents. Tests can be updated or replaced centrally (24 hours vs. 6 weeks for paper-and-pencil tests).

In addition, as reported earlier, online testing now allows the PPC to pilot new items on a continual basis at a fraction of the cost of large pilot studies (savings are estimated at around $50k to $70k annually).

Alternatives to the CGT technology used by the Program, which could further increase test security, resource usage, and enable faster testing sessions, exist. They are discussed in section 4.8 of this report.

The ability of the Program to deliver services depends heavily on the availability of information technology systems

Efficiencies reported earlier depend mostly on the availability of IT systems. As a result, IT service interruptions now have a significant impact on testing services. For example, the move of the PSC's data center to Borden, Ontario in 2015, resulted in a three week interruption to testing services. Another example is that to remedy the numerous service interruptions often associated with the Voice Over Internet Protocol telephony system, a Centrix telephony system had to be installed for all TOP assessors, as much administration of the TOP is done by phone.

In 2012, the PPC had about 17 different IT applications, some of which can no longer be supported. A need to streamline and modernize these applications was identified as a priority for the PSC. Since then, and as previously reported in section 4.4 of the report, the PSC's Information Technology Services Directorate has been leading a multi-year project to modernize, optimize and rationalize the PPC's applications.

The evaluation did not have the mandate to look at this applications rationalization project. Still, the evaluation noted through interview data collected that the Program has high expectations regarding the potential of this project to find technological efficiencies and, by extension, optimize the use of technology.

4.8 Performance: Alternatives

Are there other approaches to assessing candidates' second official language?

Other approaches to assess second official language exist but caution needs to be exercised as not all alternatives provide the assurance that official languages standards are applied uniformly across hiring organizations.

The evaluation looked at technological approaches that could improve the effectiveness of the process. It also examined how other jurisdictions were administering similar programs.

The evaluation was not mandated to look at possible alternative tests to the SLE tests currently used. To that end, a policy taskforce composed of various stakeholders, led by the PSC's Policy Branch, started to study this question at the time this evaluation was conducted.

Computer-generated adaptive tests could be beneficial for the Program in the long-term

Adaptive tests are adapted to the test taker's ability level. The SLE Test of Reading Comprehension and Test of Written Expression correspond to Linear-on-the-Fly Test (LOTF). They are non-adaptive to test takers' abilities, but are assembled for each individual immediately prior to the testing session.

Alternatives to LOTF would be:

  • Computerized Adaptive Test (CAT): Each test item is adapted in real-time to the response to its preceding question; or
  • Multi-Stage Test (MST): These tests include both non-adaptive and adaptive components. Essentially, a single MST is comprised of several sets of non-adaptive item tests, or testlets, each of which are pre constructed from calibrated items to meet precise test specifications.

Whereas a CAT is adaptive at the item level, an MST is adaptive at the testlet level. Generally, MSTs offer some flexibility over CATs, but this comes at the cost of precision. MSTs are more efficient than fixed test, but less so than fully adapted tests.

The Program's research and development unit is aware of the benefit associated with adaptive tests. It is actually looking into the possibility, on a long-term basis, of implementing adaptive components to its tests of reading comprehension and written expression.

Similar programs in other jurisdictions

A literature review was conducted to identify how other jurisdictions offering similar programs were operating. The following insert summarizes our findings in three categories.

Table
Mandatory Tests
Hiring organizations are required by a central organization to use a specific test, either developed by that organization or a third party

Examples of jurisdictions using this model are:
  • Canada (PSC)
  • United Nations (administered only once a year)
  • Belgium
Main advantages:
  • Assurance that uniform OL standards are applied across hiring organizations
  • Makes for easy portability of results across hiring processes
Main disadvantages:
  • Tests may not be completely tailored to the nature of work to be performed
  • Test administration may lack flexibility
Centralized Tests Offered but not Mandatory
Hiring organizations can choose either a common test made available to them by a central organization or use any other test they deem appropriate

Examples of jurisdictions using such a model are:
  • European Union
  • New Brunswick (but only for the oral ability; the model is fully decentralized for other abilities)
Main advantages:
  • Flexibility of approaches
  • Tests can be tailored to the nature of work to be performed
Main disadvantages:
  • Cannot assure that OL standards are applied uniformly across hiring organizations
  • Increased need for monitoring
  • Portability of results across hiring processes may be difficult
No Tests Offered
Hiring organizations can choose which tests they will use.

Examples of jurisdictions using this model are:
  • Canada (PSC) (for student and casual employment)
  • Switzerland
Main advantages:
  • Flexibility of approaches
  • Tests can be tailored to the nature of work to be performed
Main disadvantages:
  • Cannot assure that OL standards are applied uniformly across hiring organizations
  • Increased need for monitoring
  • Portability of results across hiring processes may be difficult

As discussed in the insert above, models that do not rely on mandatory tests present advantages in terms of flexibility of testing approaches, but come at the expense of ensuring that OL standards are applied uniformly across hiring organizations.

5. Conclusions

Is there a continued need for the Second Language Testing Program?

  • Second language testing is responding to a continuous need, not only for staffing purposes but also for the professional development of public service employees. This is true for Public Service Employment Act (PSEA) and non-PSEA organizations alike.

Is the Second Language Testing Program aligned with government priorities and the Public Service Commission's strategic outcome?

  • The importance of Canada's official languages is demonstrated in legislation and key strategic government documents.
  • The second language testing activities are also aligned with the Public Service Commission's (PSC) strategic outcome, as they support merit by thoroughly assessing the second official language proficiency of public servants.

Are roles and responsibilities for Second Language Testing Program clearly defined and communicated?

  • Within the public service, roles and responsibilities in relation to second official languages testing are shared between the Treasury Board, the PSC and hiring organizations.
  • Although these partners' respective roles are defined and communicated, some test takers are confused regarding who is responsible for what.

Is the Second Language Testing Program achieving its expected outcomes?

  • A majority (above 80%) of corporate users surveyed expressed satisfaction with the Second Language Evaluation (SLE) testing services and over 90% of test takers surveyed were satisfied with their testing experience.
  • For the Test of Oral Proficiency (TOP), meeting the scheduling standards has been a challenge over the last two years and user satisfaction on that level is low.
  • Dissatisfied corporate users reported having difficulty in reaching the appropriate PSC person when additional help was needed in relation to second language testing activities.
  • Many test takers are not aware of all the information available on the PSC website to help with test preparation.
  • There is a gap between the performance feedback information desired by test takers of the TOP and the information provided in the standard performance feedback they received.
  • Most test takers are aware of their accommodation rights. However, there are some who choose to waive their rights for fear of negative consequences or because they find the process of requesting accommodation too cumbersome.
  • There are adequate mechanisms to ensure continued improvement of test content and test security for all three SLE tests. Quality control monitoring for the TOP is also adequate.

Does the Second Language Testing Program demonstrating efficiency and economy?

  • Efficiency gains were made by making online testing mandatory for the reading comprehension and written expression tests. However, the Program's heavy reliance on technology makes it vulnerable to information technology system interruptions.

Are there alternative ways to deliver the Program?

  • Other approaches to assess second official language exist, but caution needs to be exercised as not all alternatives provide the assurance that official languages standards are applied uniformly across hiring organizations.

6. Management response and action plan

Management response and action plan
Recommendation Response and Planned Action Office of
Primary
Interest
Estimated
Completion
Date
Recommendation 1: The Personnel Psychology Centre (PPC)/Public Service Commission (PSC) should review its client service entry points to identify options for their optimization. Management agrees with this recommendation.

Client contact information will be reviewed and clarified / simplified where feasible. The review will examine ease of access to the information and clarity of content and include an examination of relevant documents, SLE systems (e.g. SLE-SS) and the PSC website.
PPC Nov 2016
Recommendation 2: The PPC/PSC should reassess its current resourcing model for second language assessors, including the number of FTEs needed for SLE Test of Oral Proficiency (TOP) administration, in order to be more responsive to client demand. Management agrees with this recommendation.

Management will review the recruitment model with a view to streamlining the process, exploring alternative recruitment mechanisms, examining assessment tools and screening processes and reviewing data on turnover to determine optimal resourcing levels to meet scheduling standards.
PPC Dec 2016
Recommendation 3: The PSC should review its outreach efforts and the way information on language requirements of positions and SLE tests is presented online in order to better reach its targeted audience. Management agrees with this recommendation.

Along with a review of the PSC website to optimize access to and clarity of information (see recommendation #1), an outreach strategy will be prepared to inform candidates, human resources and managers of tools and information currently available related to SLE testing.
PPC Nov 2016
Recommendation 4: The PSC should review the standard feedback it provides to TOP test takers to identify improvements to better meet the needs of test takers, while balancing the need for test security. The PSC should also study the feasibility of offering a less resource-intensive version of its detailed feedback at a lower cost. Management agrees with this recommendation.

A review of the current SLE TOP feedback form will be undertaken to make it more instructive and useful for candidates. Additional options for tailored feedback (written or in person) will also be explored.
PPC Mar 2017
Recommendation 5: The PSC should seek additional avenues to promote candidates' right to assessment accommodation and to a fair, barrier-free evaluation of their abilities. It should also review its accommodation processes to identify any potential barriers to accommodation and measures to eliminate them. Management agrees with this recommendation.

Additional opportunities and mechanisms to promote candidates' right to assessment accommodation and to a fair, barrier-free evaluation of their abilities will be pursued. A review of the assessment accommodation process will be undertaken with a view to implementing steps to streamline/ simplify the process for candidates with disabilities.
PPC Mar 2017

Appendices

Appendix 1 – Methodology and associated limitations

Methods / sources and associated limitations
Document review:
  • Background and mandate of the Program
  • Operational documents (such as test manuals, instructions, etc.)
  • Strategic organizational and governmental documents
Literature review exploring the following themes:
  • Models for test delivery and administration of large-scale standardized tests
  • Issues in the delivery and administration of tests
  • Examples of second language and large-scale testing activities outside the Canadian public service
Interviews:
  • Test takers (5)
  • Human Resources (HR) personnel (5)
  • Public Service Commission (PSC) management (7)
Associated limitations:
  • Only test takers who were employed in the public service were interviewed. The survey, however, was administered to test takers regardless of their employment status
Performance data analysis for 2010-2011 to 2014-2015:
  • All PSC systems data for Second Language Training (SLT) activities
  • Staffing activities by language requirements
  • Financial data on SLT activities
Associated limitations:
  • Lack of benchmarks, baselines and targets
  • Output data stronger than outcome data
Surveys:
  • HR personnel – 737 respondents (response rate: 43%; and confidence interval: 95% +/- 2.74%)
  • Test of Oral Proficiency test takers – exit survey (412 respondents)
  • Reading comprehension and written expression test takers – exit survey (1,515 respondents)
Associated limitations:
  • The test taker surveys were exit surveys (administered right after the test session) and, therefore, are non-probabilistic, so response rate and confidence interval cannot be reported
  • Exit surveys were chosen to avoid bias resulting from test takers having achieved (or not) the results they wanted. These surveys are non-probabilistic, so response rates and confidence intervals cannot be reported
  • To benefit from a large sample of respondents, the survey was left open for one month

Appendix 2 – Samples of performance feedback for Test of Oral Proficiency

For the two samples feedback below of contiguous levels of proficiency (B and C), the content of 4 out of 6 sections is identical (in bold). The integrity of the text of this source document is preserved as it was originally presented, including its original translation.

Appendix 2 – Samples of performance feedback for Test of Oral Proficiency
Level B Level C
General Feedback
You demonstrated a sustained ability to:
  • understand factual questions;
  • understand most concrete messages and instructions and factual discussions on the recordings;
  • be clear, most of the time, when providing a factual description of a non-routine activity, a sequence of events or a person; and
  • speak with spontaneity on concrete and non-routine topics.
You demonstrated a sustained ability to:
  • clearly understand all complex questions;
  • clearly understand speaker viewpoints and most complex information on the recording;
  • explain/describe in detail, rarely making mistakes that led to misunderstanding;
  • express and sustain an opinion, rarely making mistakes that led to misunderstanding; and
  • speak at length on complex and abstract topics.
Fluency and Extent
Your delivery was fairly consistent and natural when you discussed complex topics at length. You may have demonstrated language-related hesitations and/or self-corrections at times, but most of your hesitations were for ideas. Your delivery was fairly consistent and natural when you discussed complex topics at length. You may have demonstrated language-related hesitations and/or self-corrections at times, but most of your hesitations were for ideas.
Vocabulary Range and Precision
You demonstrated that you have sufficient vocabulary to handle concrete, non-routine situations and topics. Although you may have used some approximations, your message was clear or fairly clear when you talked about concrete, non-routine situations and topics. Your vocabulary was imprecise and interfered with the clarity of your message when you talked about more complex topics. You demonstrated that you have a broad range of vocabulary to convey nuances and talk about complex and abstract topics. You almost always had the right word to convey the exact meaning intended. Gaps in your vocabulary may have caused you to hesitate or rephrase occasionally. However, any incorrect word choices you may have made rarely hindered communication.
Grammatical Range and Control
You demonstrated enough grammatical control of simple structures to be clear in most stretches when handling concrete situations and topics. You were able to link ideas simply and distinguish clearly between events in the past, present and future. When you handled more complex topics, limitations in your grammatical range or control interfered with the clarity of your message. You demonstrated enough grammatical control of simple structures to be clear in most stretches when handling concrete situations and topics. You were able to link ideas simply and distinguish clearly between events in the past, present and future. When you handled more complex topics, limitations in your grammatical range or control interfered with the clarity of your message.
Pronunciation
Your pronunciation was clearly intelligible, even if an accent from another language was evident. Any mispronunciations were occasional and rarely interfered with communication. Your pronunciation was clearly intelligible, even if an accent from another language was evident. Any mispronunciations were occasional and rarely interfered with communication.
Comprehension
You demonstrated an ability to understand linguistically complex questions, speaker viewpoints and conversations about abstract topics. You demonstrated an ability to understand linguistically complex questions, speaker viewpoints and conversations about abstract topics.

Page details

Date modified: