COVID-19: Advice on alternative assessments for reading comprehension
Toolkit for hiring managers
- How to develop a reading test
- How to conduct the assessment
- Sample Rating Sheet
The Public Service Commission (PSC) is taking steps to provide departments and agencies with additional flexibility to staff bilingual positions while the situation related to COVID-19 is ongoing. Among other measures, deputy heads are temporarily exempted from the requirement to use the PSC’s Second Language Evaluation (SLE) tests when assessing second language proficiency (reading, writing and oral) for appointments to bilingual positions. For more information, please consult the Information for human resources specialists: Coronavirus disease (COVID-19).
To support hiring managers, the PSC has developed this toolkit. It explains how to develop a reading test to determine whether or not a candidate meets the requirements for the level being assessed (A, B or C). Since A-level positions are rather rare, the examples and tips focus primarily on the assessment of levels B and C.
In this toolkit, we provide suggestions based on a relatively simple approach where a candidate is instructed to read a text and then asked questions orally to confirm their comprehension. This serves to illustrate the basics of how to evaluate reading ability and is provided for reference only. There are many other equally legitimate ways to evaluate reading ability.
This toolkit includes the following:
- Information on how to develop a reading test
- Suggestions on how to conduct the assessment
- A sample rating sheet
Throughout we have tried to provide practical advice and sample materials that may be easily adapted to organizational needs. You may use other approaches as well, perhaps basing yourself on the requirements of the position (e.g., ask candidates to read a text and then write a summary).
How to develop a reading test
Suggestions for developing the content
- Determine what level you need to assess.
- Review the Qualification Standards in Relation to Official Languages (henceforth the Standards).
- Read the descriptions and the examples for all of the levels (A, B and C). This will help you to distinguish the level you are assessing from the other levels.
- At the B level, candidates should understand descriptive or factual material on work-related topics. They will:
- grasp the main idea of most work-related texts;
- identify specific details from the text;
- distinguish the main idea from secondary ideas.
- At the C level, candidates should understand texts dealing with a wide variety of work-related topics. They will:
- understand most complex details, inferences and fine points of meaning;
- understand material that is specialized or unfamiliar.
- Ask another person to review your reading task and the rating criteria.
- Make sure the quality is comparable in both official languages, if applicable.
- Ask someone whose second official language is the language being assessed to take the test. Then use the criteria you established to rate their response.
- Confirm that everything works as expected.
Suggestions for the test instructions and language of testing
- Provide the text to read for the assessment only in the language being evaluated, as per subsections 37(1) and 37(2) of the Public Service Employment Act.
- Make the test instructions available to candidates in English, French or in both official languages, according to their preference. Candidates must understand what is expected from them during the assessment.
- Allow candidates to answer your questions in the official language of their choice. They may choose to alternate between English and French when responding. Focus on what they say (content), and not on how they say it (language). Their ability to express themselves is not being evaluated.
Consider using a reading task similar to the following example. For this task, candidates are asked to read a text about an upcoming learning event, which is a common workplace topic. They are also asked to answer questions pertaining to the text. These
questions are to determine whether candidates can grasp the main idea or identify specific details about the text, both key elements in the Standards.
This is to remind you of an upcoming learning event about (…). As you may know, (…).
- What is the text about? Please provide me with a brief summary of the text.
- According to the text, what is the main purpose of this learning event?
- According to the text, why is the organization offering this event?
- We recommend the following minimum number of words for each level:
- Level B:1000 words
- Level C: 2000 words
- Consider asking candidates to read more than one text, covering different themes. Asking questions related to each text will help confirm candidates can understand a variety of texts. For example, at the B level, candidates could read one text of 1000 words or several texts totalling 1000 words. The same is true at the C level. Candidates could read one text of 2000 words or several texts totalling 2000 words.
- For the C level, you may find it more appropriate to use a different approach. Instead of asking questions based on one or more relatively short texts, you could ask candidates to summarize a much longer one (e.g., a whole report of 10 or more pages).
- Consider using a text that candidates would need to read in their second official language while on the job.
- Take extra care to ensure the text can be understood by all candidates (e.g., those outside your unit, outside government).
- Provide a sufficient amount of time to complete the task, without rushing the candidate. Pretesting with colleagues can be helpful to determine time limits.
- Consider whether you should prepare multiple versions of your test to avoid overuse and overexposure of your test content. This could help avoid giving any candidate an unfair advantage if the assessment is used in multiple staffing processes.
- The text and questions would be different in each version, but in all cases would need to be appropriate for the level of proficiency required by the job (A, B or C).
- Consider standardizing your procedure for all versions of your test (e.g., comparable text length, same number of questions, same time allowed).
Choosing the right text
- Decide what type of text you want candidates to read (e.g., email, report).
- Review the Standards and identify what someone at the B or C level should be able to read and understand. This will help you choose an appropriate text.
- Since the Standards are quite general, here are some criteria to consider:
- Texts at the B level:
- are about familiar and less familiar work-related topics;
- are straightforward, factual, concrete, and descriptive;
- will have a purpose like:
- informing an audience about an event or initiative;
- explaining the steps to complete a concrete, observable task;
- giving factual accounts of actions taken or events that have occurred.
- can include reference material, memoranda, reports, letters, e-mails, articles, notices, bulletins, minutes;
- do not include complex grammar, less common vocabulary and abstract ideas.
- Texts at the C level:
- are about a wide variety of work-related topics;
- may be written in a formal or informal way;
- are complex, containing detailed and/or abstract information;
- can include policy papers, research papers, technical reports, persuasive texts, administrative correspondence, contracts (service agreements);
- will have a purpose like:
- expressing an opinion or argument;
- providing information or comments on contentious issues;
- conveying abstract ideas;
- persuading an audience;
- comparing and contrasting different options, opinions, etc.
- may include commonly used specialized terms that are required for the job;
- do not include seldom-used expressions and very complex grammatical structures.
- Your options include writing a text yourself or using a text from an existing source.
- Texts can be selected from a variety of sources, such as internal documents (e.g., old e-mails, bulletins) or government websites.
- If you have the necessary time and resources, consider revising the text by:
- editing the text for length and clarity;
- modifying the theme and certain details of the text to reduce the risk of giving an advantage to someone already familiar with the text;
- replacing any identifying information (the names of people, organizations and locations) with fictitious but realistic information to protect confidentiality.
- Ensure the text:
- is appropriate for the level being assessed. This will allow candidates to demonstrate their proficiency without forcing them to read a text at a lower or higher level than the requirements of the position;
- is not unnecessarily complicated;
- is clear and free of grammatical errors, acronyms and unnecessary technical or specialized language;
- is of a type that the candidate would be expected to read on the job;
- will be understood by candidates of all backgrounds (e.g., both internal and external candidates);
- provides enough content so that you can develop a sufficient number of questions to properly assess reading comprehension;
- is in Arial 12 so it is easy to read and more accessible.
Asking the right questions
- We strongly recommended including two types of questions, no matter what level of proficiency is being assessed:
- Global Questions: Questions about the general idea of the text.
- What is this text about? Please provide me with a brief summary of the text.
- What is the purpose of this email? Briefly describe it to me.
- What are the main/key points of this text? Please provide me with a brief summary.
- Specific Questions: Questions about a specific detail within the text.
- According to this text, what is (…)?
- According to this report, how (…)?
- The author said X in the text. Why (…)?
- Global Questions: Questions about the general idea of the text.
- Consider asking candidates one global question and two or more specific questions per text.
- Ask questions that candidates can answer using the text alone.
- Avoid questions that might encourage candidates to rely on their pre-existing knowledge of a content area.
- Avoid questions that may require candidates to make assumptions or to interpret tone and nuance. Candidates may perceive them differently depending on their background.
- We have provided a sample rating sheet at the end of this document. You could use and/or adapt it.
- You may also create your own rating sheet:
- Consult the Standards.
- See the description of the B and C levels for the Test of Reading Comprehension (in the “Levels” section). This description, which was developed by the PSC and based on the Standards, presents the profile of typical performances at the B and C levels.
- Consider including the expected correct answer for each question in your rating sheet and having them validated by someone else ahead of time. Having this ready beforehand will help you rate the candidate’s performance consistently.
How to conduct the assessment
You will need to decide how to conduct the assessment. We suggest doing the following:
Before the assessment
- Review the Standards to understand what is expected at the level of proficiency required for the job (A, B or C).
- Familiarize yourself with the rating sheet you prepared for your text. This will allow you to evaluate the candidate’s performance on the spot.
- If your assessment includes multiple texts, consider preparing a distinct rating sheet for each one.
During the assessment
Administering the test
- Provide candidates with the text and the questions so they can prepare.
- Provide the text only in the language you are assessing.
- Consider providing the questions in both official languages.
- Give them adequate time to read the text.
- Consider allowing them to take notes.
- When the assessment portion begins, ask your questions orally in the candidate’s language of choice.
- Allow them to answer the questions in the language of their choice. They may switch back and forth between official languages when responding. This is fine.
- Alternatively, you may decide to ask candidates to respond to the questions in writing, in their official language of choice.
Evaluating the response
- Focus on what the candidate says (content), and not on how they say it (language). This is particularly important when the candidate has chosen to provide their answer in their second official language.
- Take note of your observations against the criteria for each text.
- If the answer is not clear, ask follow-up questions without providing the answer.
- Give credit for answers that capture the same idea as your expected correct answer, even if the wording is not the same.
- If you choose to have the candidate read multiple texts, consider evaluating one text at a time.
- Consider having more than one person perform the rating.
After the assessment
- The final result will be pass or fail:
- Meets the requirements of the assessed level (Pass)
- Includes performances that minimally meet the requirements of the level being assessed, as well as those of fully bilingual candidates.
- Does not meet the requirements of the assessed level (Fail)
- Includes performances that are slightly below the requirements, as well as those that clearly don’t meet the requirements.
- It is recommended that the benefit of the doubt be given to candidates whose performance is borderline.
- If you choose to have the candidate read multiple texts, integrate your ratings of each reading task into a final rating.
- Record any useful information for use in feedback or informal discussion sessions with candidates.
Sample Rating Sheet
|Candidate:||Key||F = Fully met expectation
P = Partially met
N = Not / Rarely met
|Criteria||Question||Expected Answer||F/P/N||Notes and Examples|
|Grasp the main idea||Q1: What is this email about? Please provide a summary.||It is about (…)|
|Identify specific details||Q2: According to the text, why (…)?||It is because …|
|Identify specific details||Q3: According to the text, (…)?||(…)|
|Meets the requirements of the required level (Pass)||☐|
|Does not meet the requirements of the required level (Fail)||☐|
|Rationale and Notes
Report a problem or mistake on this page
- Date modified: