Tool for Mitigating Biases and Barriers in Assessment

An assessment is most useful when it fully captures the qualifications required for a job. Some tools or methods may not cover the full depth, breadth and range of criteria related to a qualification. For example, using a short interview question (with no context or follow-up questions) might not be enough to assess a complex technical skill. This may cause some groups to underperform despite being qualified.

Possible mitigation strategies

Ensure the tool is valid and reliable, and that it adequately covers job-related functions.

For each qualification, develop definitions and expected levels of performance that are observable, measurable, and inclusive of different ways of working.

Consider assessing qualifications with more than one method. See the guidance on flexibility in assessment.

People can work in many different ways. Some people will demonstrate their qualifications better through one method over another (for example, by speaking versus writing). Using a single method or type of question may give fewer opportunities for people to fully demonstrate their qualifications. For instance, asking candidates to deliver an oral presentation for a position that doesn’t normally require this skill might pose a barrier to some groups.

Possible mitigation strategies

Check that the skills needed to respond during the assessment are related to the job.

Consider assessing qualifications with more than one method. See the guidance on flexibility in assessment.

Consider offering assessment options to candidates (for example, the ability to choose between equivalent questions that assess the same qualification or provide responses in their preferred format).

A sentence can be inappropriate even if it doesn’t contain any offensive language. Poor word choice can lead people to feel excluded or unclear about what is expected. For example, using government jargon can disadvantage people with less government experience.

Possible mitigation strategies

Ensure the language used in the assessment is not above the reading level required for the job. See the Canada.ca Content Style Guide for tips on plain language.

Use the Fairness Review Checklist or another tool to review material for fairness concerns.

Use inclusive language. See the Translation Bureau’s recommendations, the Inclusionary: A collection of gender-inclusive solutions and the Guide on Equity, Diversity and Inclusion Terminology.

Check that English and French versions are of equivalent quality and complexity.

Be mindful of how pronouns are used.

People cannot perform at their best if there are barriers related to technology or format. For example, the font size could be too small, or the testing platform could be difficult to navigate or not adaptable to assistive technologies.

Possible mitigation strategies

Use accessibility tools and guides, including an accessibility checker or readability levels (see the Digital Accessibility Toolkit).

Select an accessible administration platform and technology (that offers adjustable font size, that is compatible with assistive technology and can accommodate specific needs).

People perform best when they understand what they need to do, how they will be evaluated and who will be evaluating them. If the instructions are incomplete, vague or can be interpreted in different ways, people may underperform. This is especially a concern for people who are less familiar with the type of assessment method being used.

Possible mitigation strategies

Give clear instructions on how to prepare and structure responses.

Provide specific information about the assessment (what is being assessed, type and number of questions, what platform or software to use). See sample communications to candidates.

Tell candidates how many board members and who will be present or interacting with them during the assessment.

Provide ways to ask for more information (orientation session, information package, contact person).

Offer flexibilities such as change in testing date or different response formats.

People need time to think before they deliver their responses unless your goal is to assess reaction time. Insufficient preparation time can disadvantage many people or encourage them to quit the assessment process. Those who could be disadvantaged include persons with disabilities, people with limited access to computers, people who talk or type more slowly, and people with less assessment experience.

Possible mitigation strategies

Create a flexible assessment schedule, over several days.

Give ample time for people to prepare and deliver their response, considering factors relevant to the assessment method (typing speed, web navigation, response style, work or family responsibilities).

If feasible, consider providing the questions and time to prepare before meeting the assessment board.

Inconsistent administration, including different procedures, lengths, or instructions from person-to-person leaves room for error or bias. It can lead to unfair ratings despite a well-designed test, unless there is a sound rationale, such as providing assessment accommodation measures or removing a barrier for a specific group. The risk of inconsistencies grows when the number of candidates increases, when the process is long and when different people participate in the administration.

Possible mitigation strategies

Keep a consistent administration even when offering flexible options.

Create administration instructions and procedures for the board and administrators.

Provide the same information to everyone (including external candidates).

Ensure the testing environment is comfortable, accessible and has minimal distraction.

People who are less familiar with the technology used during an assessment may not fully demonstrate their qualifications. Lack of comfort using digital platforms can also raise stress and anxiety. As well, the internet and reliable technologies are not readily available in all regions and households.

Possible mitigation strategies

Offer ways to ask for more information (orientation session, information package, contact person, help desk).

Give instructions on how to use the technology for the assessment, including a demonstration, if possible. For example, what steps to follow on the day of assessment, details on how to use the technology and how to address potential issues.

Provide opportunities for people to practice using the technology before the assessment.

Sometimes people require accommodation measures to remove barriers and ensure they are tested fairly. However, the accommodation process can create barriers when it is ambiguous, complicated, or long; in these circumstances, the process and measures may not adequately meet a person’s needs.

Possible mitigation strategies

Plan how you’re going to respond to potential accommodation requests.

Notify people of their right to assessment accommodation at every stage of the assessment process.

Give people enough information about the assessment, including sample questions if possible, so they can make informed accommodation requests.

Give examples of types of assessment accommodation.

Follow guidelines such as the basics of assessment accommodation or the Guide for Assessing Persons with Disabilities. When accommodation measures are provided, ensure that they don’t change the nature and level of the qualification being assessed.

Having a structured rating method is important to minimize bias and inconsistent results. Agree on what criteria to look for and how to integrate ratings across board members.

Possible mitigation strategies

Prepare a rating booklet that describes the qualifications, rating criteria, examples of effective responses and space for taking notes.

Start by rating each qualification separately from other board members and then discuss any disagreements.

Provide a rationale for each rating given.

Consider evidence for a qualification across questions asked, as the person may have presented relevant information when answering a different question.

Select an appropriate pass mark (the minimum mark required to perform the job).

When the rating criteria are too narrow, you risk eliminating qualified people and limiting diversity. Overly restrictive criteria can disadvantage people who may respond correctly to the question but whose answer is not included in the narrow list of expected responses.

Possible mitigation strategies

Define the rating criteria broadly and include different ways to perform.

Discuss the rating criteria and how to recognize different ways to perform. Encourage diverse perspectives.

When scoring open-ended questions, there are many common rating errors that lead to inaccurate and unreliable results. Biased judgments may form from false first impressions, stereotypes, fatigue, by comparing one person to another or to oneself, by letting one good or bad answer influence the rest, and for other reasons. Board members can learn to recognize and guard against these potential rating errors to ensure fair assessments.

Possible mitigation strategies

Train the assessment board on common rating errors.

Rate each person’s response according to the rating criteria instead of how they compare to others.

Discuss the rating criteria and how to recognize different ways to perform. Encourage diverse perspectives.

Limit the number of assessments per day and leave enough time in between to reduce fatigue.

Rate responses shortly after reading or hearing them to avoid forgetting relevant information.

Board composition can influence outcomes. Board members may give more or less favourable ratings to people who look, talk, act and work like they do. The risk of bias increases when board training is limited and when assessment procedures are not well defined or planned.

Possible mitigation strategies

Assemble a board that promotes diversity and inclusion (see the Spotlight on diverse assessment boards).

Train the assessment board on common rating errors, unconscious bias and cultural sensitivity.

Check that ratings are consistent across board members, for example, by reviewing borderline cases or having regular discussions.

Too many screening questions can fatigue candidates and board members. This can lead people to withdraw their application or submit partial responses, despite having the required qualifications. It can also lead board members to administer inconsistently and apply unreliable ratings.

Possible mitigation strategies

Reduce screening questions to focus only on key requirements.

Ensure the screening questions allow for complete and relevant responses.

Using restrictive experience criteria (such as a set number of years) can pose barriers to many talented people who were previously denied work opportunities or who took family leave or medical leave. People may also interpret the criteria in different ways and choose not to apply despite being qualified.

Possible mitigation strategies

Define what you’re looking for in terms of experience and explain why it’s important to the job.

Consult widely on your proposed screening criteria to ensure they are inclusive and not preventing qualified people from proceeding to the next steps in the process.

People can learn job-related skills in many ways: from school, work, volunteering, or other life experiences. Giving preference to experience that was gained through work, within the public service or within Canada, may screen out qualified people.

Possible mitigation strategies

When asking about past behaviours, encourage people to consider all experiences, including volunteer, school or community work, within and outside Canada.

Consult others if you aren’t familiar with the type of experience described.

Education is not accessible to all. People can acquire skills in many ways beyond formal education. It may be appropriate to ask for education for technical knowledge requirements or skills that are not easily learned on the job. However, raising the education level may not be appropriate if it’s done for volume management and not directly linked to the job.

Possible mitigation strategies

Ensure qualifications are directly job-related.

Use and inform candidates of the acceptable combination of education, training, and experience in the minimum qualification standard.

Imposing strict word limits can have the unintended effect of assessing the ability to communicate, in addition to the targeted qualification. People may express themselves differently for many reasons (cultural and linguistic, for example). Restricting answers to too few words may not give everyone a fair chance to express themselves fully, and qualified candidates may be screened out.

Possible mitigation strategies

Give candidates a recommended length for the response (for example, 250 to 400 words) without imposing a fixed word limit.

If a word limit must be set, make sure it allows people to fully demonstrate their qualifications, regardless of writing style or official language used.

Knowledge of someone’s gender, race, religion, social class, age, country of origin or other personal characteristics can cause unintended biases and impact selection decisions.

Possible mitigation strategies

Avoid asking for information that is not job-related (for example, graduation date).

Consider anonymizing responses (person’s name, address, university names) by removing irrelevant personal information before scoring.

Review a sample of responses to check that scoring is applied consistently.

Some tasks might be less familiar to people with less experience in public service: for instance, preparing briefing notes for senior management or developing organizational strategies or policies.

Possible mitigation strategies

Ensure questions are directly job-related.

Give precise instructions for each task. For example, don’t assume people will understand what to include in a presentation, briefing note or memo.

Sometimes a test is more complicated than it needs to be. If a lot of mental effort is required to understand the questions, or material that is not required on the job, then this could unfairly disadvantage people.

Possible mitigation strategies

Ensure written material is simple and consistent with the mental load required on the job.

Test the questions and material with several people from a diversity of backgrounds, including job incumbents, to ensure they can be easily understood.

A written test may be an appropriate assessment method for jobs that require some level of written skills. However, some groups have oral traditions that favour sharing knowledge and experience aloud rather than in writing. This could lead some people to better demonstrate their qualifications orally than in writing.

Possible mitigation strategies

Consider offering people the option to respond using point-form notes.

Review the rating method to ensure people won’t be penalized for their written skills unless this is part of the merit criteria.

Consider offering options if written communication is not being assessed (for example, multiple-choice questions, oral assessments).

Imposing strict word limits for open-ended responses can have the unintended effect of assessing the ability to communicate, in addition to the targeted qualification. People may express themselves differently for many reasons (cultural, linguistic). Restricting the number of words may not give everyone a fair chance to express themselves fully.

Possible mitigation strategies

Give candidates a recommended length for the response (for example, 250 to 400 words) without imposing a fixed word limit.

If a word limit must be set, ensure it allows people to fully demonstrate their qualifications regardless of writing style or official language used.

People are used to using many tools and resources in their everyday work, including spell check, copy and paste, a calculator and internet resources. Restricting the use of such tools, especially if they’ll be available on the job, can disadvantage candidates and give them less time to formulate effective responses.

Possible mitigation strategies

Consider allowing the use of everyday tools (spell check, dictionary), functions (copy and paste) and resources (internet), unless they hinder the accuracy of the assessment.

Let candidates choose which tool to respond with (PowerPoint, Word, Excel, or others) when possible.

A person may be qualified for a job but unable to show this when answering questions about previous work experience. Many groups are denied work opportunities or have gained relevant experience from unpaid activities. For these reasons, focusing too heavily on past work behaviours may disadvantage qualified individuals.

Possible mitigation strategies

When asking about past behaviours, encourage people to consider all experience including volunteer, school or community work, within and outside Canada.

Select the most suitable type of questions (situational or behavioural) based on the target audience and qualifications.

Consider giving optional questions (of the same complexity and scope) that candidates can select from to best highlight their qualifications.

Asking different questions of each person can introduce biases or barriers if the questions aren’t carefully aligned with what is being assessed. Using a predetermined set of questions and follow-up questions can be more predictive of job performance and help reduce barriers related to disability, gender, and race.

Possible mitigation strategies

Give the same type of questions and options to each candidate.

Select questions and follow-up questions from an inventory of valid questions.

The interview format (in person, video, phone, individual as opposed to group) could disadvantage certain groups. For instance, phone-only interviews could disadvantage persons with disabilities related to hearing or language, or people who speak with an accent that differs from that of assessment board members. In contrast, in-person-only interviews could pose barriers for a wide variety of reasons, including poor access to transportation or travel distance.

Possible mitigation strategies

Give options for different interview formats (in person, by audio only, by audio and video, pre-recorded if relevant and available).

For phone or video interviews, make sure to limit distractions by muting microphones and using neutral or blurred virtual backgrounds.

For some people, being asked to sit facing a panel could imply hierarchy and affect their comfort and responses to the questions.

Possible mitigation strategies

Ensure the interview environment is comfortable, accessible and with minimal distraction (considering the interview format).

Set up the interview room in a way that makes each person comfortable (this could be asked of each person in advance or at the start).

Before an assessment, building rapport by using small talk can help demystify the assessment process and ensure that each candidate can fully demonstrate their qualifications. However, rapport building can introduce a bias or a barrier if handled carelessly or if inappropriate questions are asked that may lead to prohibited grounds of discrimination.

Possible mitigation strategies

Prepare introductory remarks that are informative and help put people at ease before you start the assessment.

• Tell people that pauses and nervousness are okay and don’t affect their rating, if appropriate.
• Tell people they can take a pause, restart or come back to a question.

Be mindful about using pronouns during interactions with candidates.

Sometimes a person is qualified but cannot show it because they aren’t asked enough questions or the right questions. Valid interview questions cover the full range of criteria being measured. For example, only asking one question per qualification without any follow-ups might lead to limited information and an incomplete evaluation of the qualification.

Possible mitigation strategies

Use follow-up questions to gather complete information (who, what, when, where, why, how).

Avoid questions that lead towards a desired answer.

Relying on memory, paraphrasing or summarizing responses is prone to bias and likely to amplify rating errors.

Possible mitigation strategies

Take detailed notes of what the candidate says rather than paraphrasing, and use them when rating the performance.

If candidates don’t know what the reference check will assess, it can be difficult to select suitable referees. This can result in incomplete responses that could be misinterpreted as low scores.

Possible mitigation strategies

Tell candidates how the reference check will be conducted, including the qualifications that will be assessed.

Provide recommendations on how to select referees.

People can be uneasy about asking for a reference. Sometimes there aren’t enough suitable people to ask, or the referees don’t have the opportunity to observe the target qualifications. For example, if a person was self-employed, they might withdraw their application if they are not given flexibility in who to choose for their reference.

Possible mitigation strategies

Ask for multiple referees.

Show flexibility by being open to various referees (for example, colleague, client, partner or stakeholder) that can attest to the person’s qualification.

Do not penalize a candidate if their referee is unavailable or does not follow instructions.

People may be uncomfortable asking a current manager to serve as their referee if they believe it may negatively impact their current employment. People may also be unable to ask a recent manager if they were on a prolonged leave (for medical, family or other reasons).

Possible mitigation strategies

Show flexibility by asking for any manager that can attest to the person’s qualification.

It may be hard to contact and communicate with referees abroad. A referee may not speak fluently in English or French. However, omitting this referee might lead you to miss valuable information about the person being assessed.

Possible mitigation strategies

Establish a way to contact a referee who lives abroad or who does not speak English or French and to validate the information they provide.

Plan ahead and consider a flexible approach such as informing candidates about the process and asking for referee contact information early in the assessment stage or offering options of how to collect referee’s information.

The reference check can pose many biases and barriers that are not within the control of the candidate or assessment board. The assessment board may not capture an accurate view of a person’s qualification by relying solely on a reference check; this can pose barriers if it is the only method used.

Possible mitigation strategies

Use the reference check to confirm or supplement information gathered from other assessment methods.

Consider information gathered from other tools when combined with other methods (for example, interview).

Provide another valid assessment option if reference checks pose a limitation to the candidate (see possible scenarios in the guidance on flexibility in assessment).

The reference process can be different in the public and private sectors. This can disadvantage external candidates if they’re not clearly told what to expect and how to select and prepare referees. Referees may not give enough information if expectations are unclear.

Possible mitigation strategies

Create instructions that include how to structure responses.

Provide the questions and qualifications in advance and give referees enough time to prepare their answers.

Establish a process to deal with incomplete or ambiguous responses. For example, you can request more information or clarification from the referee if necessary.

Offer different ways to respond (verbally or in writing).

Referees may give incomplete answers if they haven’t seen the target qualifications. They may also give answers that are too general, making it hard for the board to assign accurate ratings. This is more of a risk for written responses if there is no follow-up to missing information.

Possible mitigation strategies

Follow up with referees or ask for an additional referee if information is inconsistent or missing.

Use follow-up questions to gather complete information (who, what, when, where, why, how).

Avoid questions that lead towards a desired answer.

Referees might have their own personal biases and preconceptions of a good employee that can influence their evaluation. As well, a person might have been performing below expectations in their past job if their manager failed to provide adequate work accommodation measures.

Possible mitigation strategies

Focus on factual information and observed behaviour rather than generalities or impressions.

Ensure you’re assessing the candidate rather than the quality of response from their referee.

Don’t provide a rating if information is insufficient. Label the response as “unable to observe” instead of a pass or fail.

Relying on memory, paraphrasing, or summarizing responses can result in biased reporting. Taking detailed notes of what the referee said can reduce rating errors.

Possible mitigation strategies

Take detailed notes of what the referee says rather than paraphrasing.

Preconceptions about the definition of good performance and personal biases can influence the content and accuracy of the performance review. Ratings may be subjective when there is personal conflict, harassment, discrimination, inadequate work accommodation, or any bias against a specific individual or an equity-seeking group. As a result, people may feel uneasy about submitting past performance reviews, some might even not apply, and some may withdraw. Some ratings could also be inflated if there is favouritism or positive bias.

Possible mitigation strategies

Look for evidence of potential bias or contradictory information in the performance reviews.

Give people the chance to add context and explain their performance review results. For example, there might have been issues outside their control, unexpected changes to project scope, new priorities, overly complex work objectives, interpersonal conflicts or other factors.

Use another assessment method or other sources of information (work sample, interview, or other) to increase the reliability of available information and fairness of the assessment.

Practices related to performance reviews vary across managers and organizations. For example, there can be inconsistency in the level of detail provided, and the work objectives and standards can be different. Management’s expectations can also vary, even when candidates are from the same classification group or the same team. This inconsistency can limit the validity, reliability and fairness of using performance information in an appointment process.

Possible mitigation strategies

Seek clarity from the candidate or the person who wrote the performance review to obtain missing or understand inconsistent information.

Use this approach when information is relatively consistent and comparable between candidates (for example, same manager, unit or classification group).

Avoid using this approach to differentiate high versus low performers. Use it to determine whether the candidate meets or does not meet a qualification.

Tell candidates how the performance reviews will be used, including the qualifications that will be assessed and how they will be rated. Ensure they can raise any concerns related to their circumstances so that these can be addressed.

The performance review may not allow a full assessment of the qualifications, because it may be based on work objectives or workplace contexts that are different than the job being staffed. Also, it’s important to consider that a candidate behaviour in a specific work context does not always extend to other jobs.

Possible mitigation strategies

Ensure the performance review contains relevant information to assess the targeted qualifications (for example, significant overlap between the qualifications assessed in the performance review and those required for the job).

Many performance review tools within or outside the public service, include ratings for core competencies and work objectives. Managers rate performance relative to their expectations, which can vary significantly from one manager to another. Also, ratings alone may not provide enough information and may be lower for people in equity-seeking groups than for others.

Possible mitigation strategies

Consider the descriptive information during the period being assessed, focusing on demonstrated behaviours, rather than ratings.

Develop your own rating guide to systematically assess evidence of the qualifications found in the performance review.

Ask another person with relevant knowledge of the position being staffed to examine the performance review and validate its objectivity, accuracy and fairness.

People can demonstrate their qualifications in different roles and at different points in time. Some people may be qualified but only demonstrated the criteria in earlier years because of different job duties or gaps in their employment. For these reasons, not everyone will have a recent performance review, or it may not fully reflect their competencies.

Possible mitigation strategies

Allow candidates without recent performance reviews to provide older performance reviews.

Consider requesting performance reviews from multiple years to have more complete information.

Without a planned approach, the narrative assessment may not accurately measure the candidate’s qualifications. A lack of structure can introduce biases in the design, preparation and rating of a narrative assessment and it can disadvantage people belonging to equity-seeking groups.

Possible mitigation strategies

Apply a structured and systematic approach to your narrative assessment.

Give yourself enough time to gather the necessary information.

Include concrete examples of how the person has demonstrated the required qualifications.

Sometimes the written narrative description does not fully capture one or more of the qualifications being assessed. A description that lacks sufficient detail may result in an inaccurate and unfair assessment.

Possible mitigation strategies

Ensure your assessment is based on the job requirements and is not influenced by your interest in a specific person.

Ensure each qualification adequately covers current job functions and is defined with expected levels of performance that are observable, measurable and inclusive of different ways of working.

Use information from a variety of sources or from other assessments, if required (for example, work samples, interviews, reference checks).

Preconceptions about good performance or personal biases towards a specific individual or an equity-seeking group can influence the content and accuracy of a narrative assessment. When group membership is known, stereotypes may influence assessments for people belonging to equity-seeking groups.

Possible mitigation strategies

Reflect on your potential stereotypes with respect to candidates from equity-seeking groups. Review your narrative assessment for any content that reflects stereotypical adjectives or that suggests doubt regarding their capabilities, and take time to assess its impact on your assessment.

Avoid making global judgments based only on a few limited examples that may not be representative of the candidate’s typical performance.

Assess each qualification independently, using behaviour demonstrated in multiple situations.

Where appropriate, ask someone with relevant knowledge of the candidate’s performance to review the narrative assessment to validate its objectivity, accuracy and fairness.

Biases affect both the recall of specific details and our interpretation of information. When negative or positive information confirms stereotypes about equity-seeking groups, this information is more readily available in our memory and can influence the assessment.

Possible mitigation strategies

Try to gather and document as many relevant examples as possible, including both effective and ineffective behaviours. Taking the full spectrum of a person’s behaviour into consideration will allow you to minimize possible memory retrieval bias.

Ask candidates to provide any relevant additional information to ensure that all important achievements are considered.