Advice on alternative assessments for oral proficiency
Toolkit for hiring managers
Table of contents
- Introduction
- How to develop the assessment content
- How to assess candidates
- Demystifying Levels B and C
- Sample Note-Taking Sheet for Interviewers (Level B)
- Sample Note-Taking Sheet for Interviewers (Level C)
- Sample Scoring Sheet for Interviewers
- Example Interview for Level B
- Example Interview for Level C
Introduction
The Public Service Commission (PSC) provides authorized departments and agencies with additional flexibility to staff bilingual positions by allowing them to use alternative methods for second language evaluation.
To support hiring managers with the assessment of oral proficiency, the PSC has developed this toolkit. It includes the following:
- Information on how to develop the assessment content
- Tips on how to assess candidates
- Sample interviews
The suggestions and examples provided in the toolkit are for reference only. Interviewers can use other strategies and methods, which are equally valid.
Since A-level positions are rather rare, the examples and tips focus primarily on the assessment of levels B and C.
How to develop the assessment content
Assessment method
- The assessment can be conducted using any method, as long as it lasts at least 10 minutes for level B and 15 minutes for level C. This will ensure that a sufficient linguistic sample is obtained to properly assess the candidate.
- Note-taking by candidates is left to the discretion of hiring managers.
- The use of semi-structured interviews is recommended, but is not mandatory.
- Suggested steps for semi-structured interviews:
- Use the same interview protocol for all candidates in the same staffing process;
- Ask candidates questions that are equivalent;
- Ask follow-up questions adapted to each candidate, based on a set protocol;
- Avoid using the same questions for a large number of candidates in order to maintain the integrity of the assessment content.
- Use a standardized method to assess responses against the predetermined criteria.
- Suggested steps for semi-structured interviews:
Suggestions for developing the assessment
1. Review the Qualification Standards in Relation to Official Languages.
- Read the descriptions and the examples for all of the levels (A, B and C). This will help you to distinguish the level you are assessing from the other levels.
2. Determine the areas to be assessed
- At the B Level, the candidate should be able to:
- Sustain a conversation on concrete topics;
- Report on actions taken;
- Provide factual descriptions and explanations;
- Give factual accounts of actions taken or events that have occurred;
- Deliver presentations on concrete topics and answer factual follow-up questions;
- Etc.
- At the C Level, the candidate should be able to:
- Support opinions;
- Express hypothetical and conditional ideas;
- Give explanations involving complicated details, hypothetical questions, or complex and abstract ideas;
- Deal with situations requiring persuasion/negotiation and complex arguments;
- Counsel and give advice to employees or clients on sensitive or complex issues;
- Etc.
- Identify aspects and activities that correspond the most to the activities the incumbent of the position to be staffed will be expected to carry out. These aspects and activities will be assessed.
- Ensure that the aspects and activities assessed are sufficiently different to confirm that candidates can use their second official language to interact in a variety of situations.
3. Develop the content for the assessment
- Refer to the "Developing Interview Questions" section below to develop questions or prompts.
- Consider a range of options, including the following:
- Develop generic questions applicable to all candidates;
- Develop questions based on the candidate's past experience.
- Ask another person to review the assessment content.
- Make sure that the content is of comparable quality in both official languages, if applicable.
- Test the content beforehand to ensure that the desired results are obtained.
Developing Interview Questions
General guidelines (for all levels)
- Formulate questions that target the appropriate level of difficulty. This will allow candidates to demonstrate their proficiency without forcing them to answer questions at a lower or higher level than the requirements of the position.
- For each topic, develop a main question and follow-up questions.
- Be clear and concise.
- It is recommended that you read the questions aloud; they should be easy to ask aloud during the oral test.
- Avoid questions that are unnecessarily complicated. Someone whose first official language is the language being assessed should be able to understand and answer the questions.
- Formulate questions that give candidates the opportunity to develop answers made up of a few consecutive sentences. To do this, use open-ended questions and limit the use of closed-ended or choice questions. Open-ended questions start with a question word such as “how” and “why” and require more than a simple “yes” or “no” answer. They make it possible to obtain a more complete linguistic sample.
Examples of open-ended questions (that can be used):- What were the project steps?
- How do you (...)?
- How did you react when (...)?
- Was planning the first step of this project? (closed-ended question)
- Did you carry out this project alone or in a team? (choice question)
- Do you have to do X before doing Y? (closed-ended question)
- Were you satisfied or dissatisfied with the results? (choice question)
- Don’t present topics that might be potentially upsetting to candidates (e.g., politics, religion, negative experiences or situations).
- Formulate questions to assess the ability of candidates to use their second official language, and not to assess their knowledge of a specific subject.
- Formulate questions that are at the level being assessed, using the appropriate grammar and vocabulary. For example, avoid using specialized terms or expressions.
Do:- Why was this client angry?
- What were the project steps?
- Why was this client irate?
- What project completion milestones did you, your team and business partners reach to bring this project to fruition?
Level B questions
- Develop questions that deal with concrete and factual job-related topics (questions about facts or procedures, such as steps, required information, etc.).
- Include questions that will help to determine the candidate’s ability to situate events in time (past, present, future).
- Avoid questions that are too simple, leading candidates to discuss very familiar and routine topics (Level A).
- Some candidates will simply provide short answers. They may be able to do more.
- Other candidates will provide longer, but simple responses. This can give a false impression that they are more proficient than they actually are.
- Examples of questions that are too simple:
- Who does / did X?
- Where do you work?
- When did you do X?
- Avoid overly complex questions, leading candidates to provide detailed explanations, express their opinion or discuss hypothetical topics (Level C). Normally, B-level candidates would have difficulty answering these types of questions, which may cause them unnecessary stress, while not adding useful information to the assessment. Please consult the next section for examples of C-level questions.
- Watch the following videos: How to Assess the B Level and PSA-B Interview Tips. They contain important information on how to assess the B level.
Level C questions
- Develop one or two exploratory questions. These questions will help you to identify topics that you can further explore to obtain detailed explanations.
- Develop several questions that will lead candidates to provide detailed explanations, support their opinionFootnoteE1 or even discuss hypothetical topics. Make sure that you vary the types of questions.
Examples:- What were the impacts of X on Y? (detailed explanations)
- What would you do if (…)? (hypothetical question)
- What are your thoughts on (…)? (opinion question)
- Include questions that assess the candidate’s ability to use precise vocabulary and a broad range of grammatical structures and verb tenses.
- Avoid questions that are too simple, leading candidates to discuss very familiar and routine topics (Level A).
How to assess candidates
Before the assessment
- Review the Qualification Standards in Relation to Official Languages to understand what is expected at the assessed level (A, B or C).
During the assessment
- Assess candidates against the standard being assessed. Do not compare candidates to one another.
- Assess the clarity of the language used to convey the message, and not the content of the responses.
- Consider both strengths and weaknesses.
- Determine if the candidate can consistently sustain a performance that is equal to or higher than the level being assessed.
After the assessment
- Integrate the results to get a final score (if there are multiple interviewers).
- Award a final result:
- Meets the requirements of the assessed level (Pass)
- Includes performances that minimally meet the requirements of the level being assessed, as well as those of fully bilingual candidates.
- Does not meet the requirements of the assessed level (Fail)
- Includes performances that are slightly below the requirements, as well as those that clearly don’t meet the requirements.
- Record any useful information for use in feedback or informal discussion sessions with candidates.
Demystifying Levels B and C
Here are some tips to better understand what is expected at the B and C levels:
- Review the descriptions for levels A, B and C in the Qualification Standards in Relation to Official Languages to have a better understanding of all three levels.
- See the descriptions of the B and C levels for the Oral Language Assessment (in the “Levels” section). These descriptions, developed by the PSC and based on the Qualification Standards in Relation to Official Languages, present the profile of typical performances at the B and C levels.
Sample Note-Taking Sheet for Interviewers (Level B)
Candidate: | Key | C = Consistently demonstrates competence O = Occasionally demonstrates competence D = Does not / Rarely demonstrates competence |
Interviewer: | ||
Date: |
Behavioural Indicators | C / O / D | Notes and Examples |
Understands questions and statements spoken in a standard dialect that deal with concrete, work-related topics delivered at normal speed. | ||
Speaks with spontaneity, although pauses for grammatical and lexical planning and repair are evident in longer stretches. | ||
Uses appropriate vocabulary to discuss concrete, non-routine situations and topics. There may be vocabulary difficulties in some areas, but most stretches are clear. | ||
Uses a variety of simple grammatical structures to discuss concrete, non-routine situations and topics. Situates events in time (past, present, future) and links a series of simple elements. There may be grammar difficulties in some areas, but most stretches are clear. | ||
Pronunciation is generally clear enough to be understood. | ||
General Comments:
|
Sample Note-Taking Sheet for Interviewers (Level C)
Candidate: | Key | C = Consistently demonstrates competence O = Occasionally demonstrates competence D = Does not / Rarely demonstrates competence |
Interviewer: | ||
Date: |
Behavioural Indicators | C / O / D | Notes and Examples |
Understands linguistically complex speech that deals with work-related topics and is spoken in standard dialect at normal speed. | ||
Has a fairly natural and even delivery, with occasional hesitations, but most hesitations are for ideas (which is normal, even in the first official language). A person at this level will not have the ease and fluency of a native speaker. | ||
Uses a broad range of vocabulary when talking about complex, abstract and hypothetical topics, with a relatively high degree of control. Errors rarely lead to misunderstanding. | ||
Uses a broad range of grammatical structures and verb tenses when talking about complex, abstract and hypothetical topics, with a relatively high degree of control. Errors rarely lead to misunderstanding. | ||
Pronunciation is generally very clear. | ||
General Comments:
|
Sample Scoring Sheet for Interviewers
Candidate: | Date: |
Interviewer: | Total assessment time: |
Questions: 1- 2- 3- … |
|
Standard Description (List of aspects assessed taken from the Qualification Standards in Relation to Official Languages.) |
Final Score Required Level: |
B ☐ C ☐ |
Meets the requirements of the required level (Pass) | ☐ |
Does not meet the requirements of the required level (Fail) | ☐ |
Rationale and Notes
|
Example Interview for Level B
A. Ask exploratory work-related questions. Something like:
- What are your main tasks at work?
- What interests you most about this position?
- What major projects did you work on at school?
B. Ask your 1st question. Something like:
- What are the steps to do X?
- How do you (...)?
- What do you do when (...)?
Ask follow-up questions.
C. Ask your 2nd question. Something like:
- Please explain Y.
- Please describe Y.
Ask follow-up questions.
D. Ask your 3rd question (past event). Something like:
- On your résumé, you mentioned that you worked on Z. Please describe what you did.
- How did you react when (...)?
- Why was the client happy?
Ask follow-up questions.
E. Ask your 4th question (future event). Something like:
- Explain what your priorities will be at work over the next few weeks.
- Tell me about the work-related courses that you will be taking this year.
- What are your career objectives?
Ask follow-up questions.
Tips:
- Ask open-ended questions to encourage longer responses.
- Choose concrete or factual topics.
- Make sure that the assessment lasts for at least 10 minutes and that the candidate does most of the talking during those 10 minutes.
Example Interview for Level C
A. Ask one or two exploratory work-related questions to identify topics that could be expanded upon. Something like:
- What are your main tasks at work?
- What are the steps to do X?
- On your résumé, you mentioned that you worked on Z. Please describe what you did.
B. Ask a question to get detailed explanations. Something like:
- Explain how/why (complex aspect of the candidate's work).
- What were the impacts of X on Y?
Ask follow-up questions.
C. Ask the candidate to talk about a hypothetical situation. Something like:
- What would you do if (...)?
- What would have happened if (...)?
Ask follow-up questions.
D. Ask the candidate to give and sustain their opinion. Something like:
- What are your thoughts on (...)? Why?
- In your opinion, what makes a good employee: x or y? Why?
- In your opinion, is there any real advantage to (...)? Why?
Ask follow-up questions.
Tips:
- Ask open-ended questions to encourage longer responses.
- Choose complex, abstract and hypothetical subjects.
- Make sure that the assessment lasts for at least 15 minutes and that the candidate does most of the talking during those 15 minutes. In some cases, the performance could be uneven. When this occurs, the assessment will need to last a bit longer to determine if the candidate can consistently sustain their performance at the level being assessed.
Page details
- Date modified: