Algorithmic Impact Assessment Tool

On this page

1. Introduction

The Algorithmic Impact Assessment (AIA) is a mandatory risk assessment tool intended to support the Treasury Board’s Directive on Automated Decision-Making (“the Directive”). The tool is a questionnaire that determines the impact level of an automated decision-system. It is composed of 48 risk and 33 mitigation questions. Assessment scores are based on many factors including systems design, algorithm, decision type, impact and data.

The AIA was developed based on best practices in consultation with both internal and external stakeholders. It was developed in the open, and is available to the public for sharing and re-use under an open license.

2. Using and Scoring the Assessment

This assessment is organized according to Government of Canada’s policy, ethical, and administrative law considerations of automated decision system risk areas as established through the Treasury Board of Canada Secretariat’s consultations with academia, civil society, and other public institutions. The AIA is designed to help departments and agencies better understand and manage the risks associated with automated decision systems. The AIA is composed of questions in various formats to assess the areas of risk defined in Table 1.

Table 1. Risk Area Definitions
Risk Areas Definition
1. Project
Project Phase Project owner, description and stage (design or implementation).
Business Drivers / Positive Impacts Motivation for introducing automation into the decision-making process.
Risk Profile High-level risk indicators for the project.
Project Authority Need to seek new policy authority for the project.
2. System
About the System Capabilities of the system (e.g., image recognition, risk assessment).
3. Algorithm
About the Algorithm Transparency of the algorithm, whether it is easily explained.
4. Decision
About the Decision Classification of the decision being automated (e.g., health services, social assistance, licensing).
5. Impact
Impact Assessment Duration, reversibility and area impacted (freedom, health, economy or environment).
6. Data
Source Provenance and security classification of data used to automate decisions.
Type Nature of the data used as structured or unstructured (audio, text, image or video).

The AIA also assesses the mitigation measures in place to manage the risks identified. These mitigation questions are organized into the categories defined in Table 2.

Table 2. Mitigation Area Definitions
Mitigation Areas Definition
7. Consultation
Internal and External Stakeholders Internal and external stakeholders consulted, such as privacy and legal experts.
8. De-Risking and Mitigation Measures
Data Quality Processes to ensure that data is representative and unbiased, as well as transparency measures related to those processes.
Procedural Fairness Procedures to audit the system and its decisions, as well as the recourse process.
Privacy Measures to safeguard personal information.

2.1 Scoring

Each area contains 1 or more questions, and the responses to the questions contribute to a maximum score for the area. The value of each question is weighted based on raw impact and mitigation, and contributes to the score based on this weighting. The raw impact score measures the risks of the automation, while the mitigation score measures how the risks of automation are managed.

The questions in risk areas 1 to 6 increase the raw impact score, and the questions in mitigation areas 7 and 8 increase the mitigation score. The questions are designed to measure the impact that decisions will have across a broad range of factors, including:

  • the rights of individuals or communities,
  • the health or well-being of individuals or communities,
  • the economic interests of individuals, entities, or communities,
  • the ongoing sustainability of an ecosystem, and
  • the duration and reversibility of the impacts.
Table 3. Raw impact score from the Risk Areas
Risk Area No. of Questions Maximum Score
1 - Project 15 15
2 - System 1 0
3 - Algorithm 2 6
4 - Decision 1 6
5 - Impact 16 36
6 - Data 13 44
Raw impact score 48 questions 107 maximum
Table 4. Mitigation score from the Mitigation Areas
Mitigation Area No. of Questions Maximum Score
7 - Consultations 2 2
8 - De-Risking and Mitigation Measures 31 43
Mitigation score 33 questions 45 maximum

The current score is determined as follows:

  1. if the mitigation score is less than 80% of the maximum attainable mitigation score, the current score is equal to the raw impact score; or
  2. if the mitigation score is 80% or more of the maximum attainable mitigation score, 15% is deducted from the raw impact score.

Expressed as a percentage range, this current score fits within a range corresponding to an impact level as shown in Table 4.

2.2 Impact Levels

The completion of the AIA provides the impact level for the system in question. The impact levels range from Level I (little impact) to Level IV (very high impact). The impact level is determined by the percentage of the current score against the maximum attainable raw impact score.

Table 5. Impact Level Definitions
Impact Level Definition Score Percentage Range
Level I Little to no impact 0% to 25%
Level II Moderate impact 26% to 50%
Level III High impact 51% to 75%
Level IV Very high impact 76% to 100%

3. Instructions

The AIA is available as an online questionnaire on the Open Government Portal. When the questionnaire is completed, the results provide an impact level and a link to the requirements under the Directive. The detailed results page will also explain why the system was rated a certain level. The results and the explanation can be printed or saved as a PDF.

The AIA assesses automated decisions on a broad range of topics including service recipients, business processes, data, and system design decisions. It is best to complete the AIA with a multi-disciplinary team that brings expertise for all of these areas.

Each question in the AIA must be answered. If the answer to a question is unknown, please select the lowest score option for the question. Where applicable, be prepared to provide the documentary evidence upon request.

Section 6 of the Directive provides a comprehensive list of the requirements that departments are responsible for. Some of the requirements increase for higher impact levels, including the type of peer review and the extent of human involvement in the decisions. For a complete list of requirements which vary with the impact level, refer to the appendix Impact Level Requirements in the Directive. Other requirements are baseline requirements that do not vary according to the impact level, such as consulting with the institution’s legal services prior to the development of the system, training employees and providing applicable recourse options to challenge the decisions.

3.1 When to Complete the AIA

The AIA should be completed at the beginning of the design phase of a project. The results of the AIA will guide the mitigation and consultation requirements to be met during the implementation of the automated decision system as per the Directive.

The AIA should be completed a second time, prior to the production of the system, to validate that the results accurately reflect the system that was built. The revised AIA should be released on the Open Government Portal as the final results.

3.2 Releasing the Results

Departments are responsible for releasing the final results of the AIA in an accessible format and in both official languages on the Open Government portal. The results page of the AIA provides the option to provide translations for the text entered in the AIA. The results page also provides the option to download the results as an accessible PDF to meet this requirement.

Report a problem or mistake on this page
Please select all that apply:

Thank you for your help!

You will not receive a reply. For enquiries, contact us.

Date modified: