COVID-19: Advice on alternative assessments for written expression

Toolkit for hiring managers

Table of Contents

Introduction

The Public Service Commission (PSC) is taking steps to provide departments and agencies with additional flexibility to staff bilingual positions while the situation related to COVID-19 is ongoing. Among other measures, deputy heads are temporarily exempted from the requirement to use the PSC’s Second Language Evaluation (SLE) tests when assessing second language proficiency (reading, writing and oral) for appointments to bilingual positions. For more information, please consult the Information for human resources specialists: Coronavirus disease (COVID-19).

To support hiring managers, the PSC has developed this toolkit. It explains how to develop a writing test to determine whether or not a candidate meets the requirements for the level being assessed (A, B or C). Since A-level positions are rather rare, the examples and tips focus primarily on the assessment of levels B and C.

In this toolkit, we provide suggestions based on a relatively simple approach where a candidate is asked to write an email based on a scenario created by the hiring manager. This serves to illustrate the basics of how to evaluate writing skills and is provided for reference only. There are many other equally legitimate ways to evaluate writing skills.

This toolkit includes the following:

  • Information on how to develop the writing test
  • Suggestions on how to rate the test
  • Sample writing tasks
  • Sample rating grids

Throughout we have tried to provide practical advice and sample materials that may be easily adapted to organizational needs. You may use other approaches as well, perhaps basing yourself on the requirements of the position (e.g., ask a candidate to prepare a briefing note or a short report).

How to develop the writing test

Suggestions for developing the content

  1. Determine what level you need to assess.
  2. Review the Qualification Standards in Relation to Official Languages (henceforth the Standards).
    • Read the descriptions and the examples for all of the levels (A, B and C). This will help you to distinguish the level you are assessing from the other levels.
  3. Determine what kinds of things a candidate at the required level of proficiency should be able to write. Choose one that makes sense given the required level.
    • At the B level, a candidate should be able to write a descriptive, factual text.
    • At the C level, a candidate should be able to provide a detailed explanation about something complex.
  4. Consult the Standards to learn what is expected regarding the quality of the language. Use this information to create the rating criteria.
    • At the B level, the grammar and vocabulary will require some correction.
    • At the C level, the grammar and vocabulary should be generally appropriate, with few corrections required.
  5. Try out the test material before using it.
    • Ask another person to review your writing task and the rating criteria.
    • Make sure the quality is comparable in both official languages, if applicable.
    • Ask someone whose second official language is the language being assessed to take the test. Then use the criteria you established to rate their response.
    • Confirm that everything works as expected.

Suggestions for the test instructions and language of testing

  • Make the test instructions available to candidates in English, French or in both official languages, according to their preference. Candidates must understand what is expected from them during the assessment.
  • Include information about the evaluation criteria to let candidates know how their text will be evaluated.
  • Candidates must write their response in the language being evaluated, as per subsections 37(1) and 37(2) of the Public Service Employment Act.

Writing task

Consider using a writing task that is setup like the following example. Here a candidate is asked to write an email to a team member, which is a common workplace task. The candidate is given topics to write about. They are also explicitly instructed to do things like “describe” and “explain,” which you will find in the Standards.

Example:
(Topic and scenario)
One of your team members, Patrick, would like to know (…). Write him an email about X and Y.

(Prompts)
In your email, you must:

  • state the purpose of the email
  • describe (…)
  • explain (…)

Tips:

  • Consider giving candidates the option of choosing from at least two topics to write about. This will help ensure they will find one topic that is suitable for them.
  • Allow them to base their text on their own experience or knowledge. This will minimize the risk of it becoming a creative writing exercise.
  • Make sure that you get a sufficient writing sample to do your evaluation. This should include explicit instructions to candidates regarding the minimum and maximum length of the text. Specifying a range allows for differences in writing style across individuals and between the languages (French texts tend to be a bit longer). We strongly recommend using the following length ranges:
    • Level B: 400-500 words.
    • Level C: 500-600 words.
  • Provide a sufficient amount of time to complete the task, without rushing the candidate (e.g., 90 minutes).
  • Consider instructing candidates to use complete sentences only. Point form or bulleted lists will make it difficult to evaluate their mastery of grammar.

Topic and scenario

  • Review the Standards and identify what someone at the B or C level should be able to do.
  • Decide what type of text you want candidates to write (e.g., email).
  • Create a brief scenario based on a topic of your choice (e.g., 45 – 55 words).
    • Most writing is done with a purpose and audience in mind. The scenario will help candidates to situate themselves in the given context and to formulate ideas about what to write about.
    • Consider using a realistic scenario where an employee would be expected to write in their second official language on the job (e.g., client, employee, member of the public).
    • You may use a similar scenario for both B and C tests. However, the writing task and the prompts must reflect what is expected at that specific level. To find out if candidates can perform at the B level, you must ask them to perform a B-level task. The same is true at the C level.

Tips:

  • Use plain language.
  • Make sure that the scenario and topics will be understood by all candidates (e.g., both external and internal candidates).
  • Avoid unnecessarily complicated topics or scenarios, or those that force candidates to be creative or to come up with ideas.
  • Ensure the topics or scenarios assess the ability of candidates to use their second official language, and not their knowledge of a specific subject.
  • Make it clear to candidates that they can assume the person they are writing to is familiar with their field. This will keep their focus on the writing task itself, and not on sharing their knowledge of a specific subject.

Prompts

  • Prompts tell candidates what elements they must include in their response. These specific instructions come right after the scenario.
  • Choose 2 or 3 prompts based on what candidates at the B or C level should be able to do (see the Standards).
  • Ensure the prompts are appropriate for the level being assessed. Avoid using prompts that are too easy or too difficult. This will allow candidates to demonstrate their proficiency without forcing them to produce a text at a lower or higher level than the requirements of the position.

Tips:

  • Explicitly mention that candidates must respond to the prompts.
  • Formulate prompts that give candidates the opportunity to develop answers made up of consecutive sentences.
    • For the B level, avoid prompts that lead candidates to provide detailed explanations, express their opinion, or deal with hypothetical situations (C level). These behaviours exceed the requirements of the B level. Consequently, B-level candidates will usually have difficulty answering such questions. This will not add useful information to your assessment and may cause candidates unnecessary stress.
    • For the C level, avoid prompts that are too simple, concrete and factual. Those types of prompts are more suitable at the A and B levels.

Material and equipment

  • Allow candidates to use a computer to complete the writing task.
    • Consider letting them use their own personal keyboard.
    • Give them time before the test begins to try the keyboard and mouse they are expected to use (e.g., type a few sentences to get familiar with the keyboard layout).
  • Consider allowing candidates to use the built-in spell check and grammar check functions.
    • Employees use such tools on the job when writing in either official language.
    • A high degree of language proficiency is required to use such tools effectively because these tools miss many kinds of errors and often make inappropriate suggestions.
  • Allow them to make notes.

How to develop the evaluation criteria

  • We have provided sample rating grids for the B and C levels at the end of this document. You could use and/or adapt them.
  • You may also create your own list of criteria.
    • Consult the Standards.
    • See the description of the B and C levels for the Test of Written Expression (in the “Levels” section). These descriptions, which were developed by the PSC and based on the Standards, presents the profile of typical performances at the B and C levels.
    • Ensure that your criteria are observable.
    • We recommend evaluating whether or not the text includes a response for each prompt. This is called “text completeness" in the sample rating grids.
    • We also recommend including criteria such as grammar and word choice, given that it is a language test.

How to rate the response

Rating process

  • The following two results are the only results possible, regardless of the assessment method used:
    • Meets the requirements of the assessed level (Pass)
      • Includes performances that minimally meet the requirements of the level being assessed, as well as those of fully bilingual candidates.
    • Does not meet the requirements of the assessed level (Fail)
      • Includes performances that are slightly below the requirements, as well as those that clearly don’t meet the requirements.
  • Determine the rating process you will use. A sample rating process can be found at the end of this document.
    • Consider having more than one rater.
    • If there are 2 or more raters, the final result will be obtained by integrating the results they arrived at independently.
    • The rating can be done in several stages.

Tips

  • Determine if the candidate can consistently sustain a performance at the given level or higher.
  • Rate against the Standards. Do not compare candidates to one another.
  • Assess the clarity of the language used.
  • Consider both strengths and weaknesses.
  • Give the benefit of the doubt to candidates whose performance is borderline (between “meet” and “does not meet”).
  • Do not evaluate the content of the response. The purpose of the test is not to assess their knowledge of a specific subject.
  • Only count an error once, even if:
    • the same one is made several times (e.g., “assure” instead of “ensure”).
    • it fits under more than one evaluation criteria.
      • Count the error against the evaluation criterion in your list that seems the most relevant.
      • Example: (weather/whether) "Weather he comes or not" could be considered a word-choice error or a problem with how the text flows.
  • Record any useful information for use in feedback or informal discussion sessions with candidates.

Sample Writing Task (Level B)

  • You have 90 minutes to complete the test (plan, write and edit).
  • You will use a computer.
  • You may use the built-in spell check and grammar check functions.
  • Write a response that is a minimum of 400 words and a maximum of 500 words.
  • Use complete sentences. Do not use bullets.
  • You may use headings.
  • Base yourself on your own experience or knowledge.
  • You may make notes; however, you must destroy them at the end of the test session.

You and Juliana have the same educational background. She works for a different organization. You received an email from her because she wants to apply for the same type of job that you have. She wants to know about your work and how you obtained your current job. Respond to her email. You can assume she understands the work-related terminology for your job.

In your reply, you must:

  1. briefly state the purpose of the email
  2. describe what you do at work
  3. describe how you obtained your current job

The purpose of prompts 2 and 3 is to determine if the candidate can write about factual, concrete information in the past and present (Level B). The order of these prompts can be changed. Note that this example is most appropriate for candidates with prior work experience.

Sample Writing Task (Level C)

  • You have 90 minutes to complete the test (plan, write and edit).
  • You will use a computer.
  • You may use the built-in spell check and grammar check functions.
  • Write a response that is a minimum of 500 words and a maximum of 600 words.
  • Use complete sentences. Do not use bullets.  
  • You may use headings.
  • Base yourself on your own experience or knowledge.
  • You may make notes; however, you must destroy them at the end of the test session.

You and Juliana have the same educational background. She works for a different organization. You received an email from her because she saw a job posting in your organization for the same type of job that you have. She is thinking about applying, but she is unsure. You feel that she should apply. She would like to know about your work in detail. Respond to her email. You can assume that she understands the work-related terminology you use in your job.

In your reply, you must:

  1. briefly state the purpose of the email
  2. describe what you do at work
  3. choose one task or project and describe it in detail  
  4. explain in depth why you would highly recommend that Juliana apply for the position

The purpose of prompts 3 and 4 is to determine if the candidate can describe something in detail (Level C) and use appropriate hypothetical sentences to provide advice (Level C). Note that this example is most appropriate for candidates with prior work experience.

Sample Rating Process

Initial rating

  • Familiarize yourself with the entire text by reading it through, without making any corrections or taking any notes. This first reading should give you an idea of whether or not the candidate has the proficiency level required for the job (based on the Standards).
  • Before moving on to the next step, take note of whether you think the candidate "meets" or "does not meet" the language requirements. This might be easy or challenging depending on a variety of factors, but make the best judgement you can.

Rating of the criteria

  • Read the text more carefully to rate the criteria that you established.
    • "Text completeness" criterion:
      • Check that the candidate’s response includes all the required elements that were specified in the prompts for the writing task.
      • Do not evaluate any other criteria at this stage. The candidate should get credit for completing this criterion even if there are language issues.
    • Other criteria
      • Look at the criteria and write down your observations (e.g., consistency of the performance for each criterion, the types of errors made, what was done well).

Final result (obtained independently)

  • Compare your rating of the specific criteria to your initial evaluation. If there is a discrepancy, try to understand why and try to resolve the problem. Here are a few questions to help you:
    • Did I focus too much on the errors?
    • Did I overanalyse the text?
    • Did I have to think about or guess what the candidate was trying to say?
    • Was I too generous with my initial evaluation?

Integration of results

  • When there are two or more raters, they must discuss their evaluation and determine the final result. In most instances, raters will be able to achieve consensus. However, in rare instances when they cannot, we suggest they seek input from another independent rater.

Sample Rating Grid (Level B)


Candidate:
Key C = Consistently demonstrates competence
O = Occasionally demonstrates competence
D = Does not / Rarely demonstrates competence
Rater:
Date:

Final result:
☐ Meets the requirements of level B
☐ Does not meet the requirements of level B

Evaluation Criteria C / O / D Notes and Examples
Text completeness: The text includes a response for each prompt.    
Text flow: The information and ideas in the text are often presented in the appropriate order. The reader does not have to make much effort to follow the writer’s line of thought. However, there may be some gaps in the flow of ideas that can cause occasional confusion, forcing the reader to fill in the gaps.    
Word choice: Most of the words used in the text are adequate. There is some variety in the words used, so the same words are not used repeatedly. There may be some incorrect words or some repetition, but the overall message is clear enough. The reader may have to make some assumptions.    
Grammar (sentence structure, verb tenses, word order, etc.): The text shows some variety in the use of grammatical structures, and these tend to be used appropriately. There may be some grammatical errors, but the information is conveyed clearly enough. The reader may have to make some assumptions.    

General Comments:

 

 

 

Final result:
☐ Meets the requirements of level B
☐ Does not meet the requirements of level B

Sample Rating Grid (Level C)


Candidate:
Key C = Consistently demonstrates competence
O = Occasionally demonstrates competence
= Does not / Rarely demonstrates competence
Rater:
Date:

Final result:
☐ Meets the requirements of level C
☐ Does not meet the requirements of level C

Evaluation Criteria C / O / D Notes and Examples
Text completeness: The text includes a complete response for each prompt.    
Text flow: The information and ideas in the text are almost always presented in the appropriate order. Gaps in the flow of ideas are very infrequent and cause little, if any, confusion. The reader can almost always follow the writer’s line of thought.    
Word choice: The words used in the text are almost always correct and precise. A wide variety of words are used. Few incorrect words are used and there is little repetition. The impact on the clarity of the message is almost always insignificant. The reader rarely has to make assumptions.    
Grammar (sentence structure, verb tenses, word order, etc.): A wide variety of grammatical structures are used in the text, and they are almost always used appropriately. There are few grammatical errors and the impact on the clarity of the message is almost always insignificant. The reader rarely has to make assumptions.    

General Comments:

 

 

 

Final result:
☐ Meets the requirements of level C
☐ Does not meet the requirements of level C

Sample Scoring Sheet for Raters


Candidate:
Rater 1:
Date: Rater 2:

Topic chosen:

 

Standard Description

(List of aspects assessed taken from the Qualification Standards in Relation to Official Languages.)

Final Score
Required Level:

 

B ☐
C ☐

Meets the requirements of the required level (Pass)
Does not meet the requirements of the required level (Fail)

Rationale and Notes

 

 

 

 

Report a problem or mistake on this page
Please select all that apply:

Thank you for your help!

You will not receive a reply. For enquiries, contact us.

Date modified: