Algorithmic Impact Assessment Tool

1. Introduction

The Algorithmic Impact Assessment (AIA) is a mandatory risk assessment tool intended to support the Treasury Board’s Directive on Automated Decision-Making (“the Directive”). The tool is a questionnaire that determines the impact level of an automated decision-system. It is composed of 48 risk and 33 mitigation questions. Assessment scores are based on many factors including systems design, algorithm, decision type, impact and data.

The AIA was developed based on best practices in consultation with both internal and external stakeholders. It was developed in the open, and is available to the public for sharing and re-use under an open license.

2. Using and Scoring the Assessment

This assessment is organized according to Government of Canada’s policy, ethical, and administrative law considerations of automated decision system risk areas as established through the Treasury Board of Canada Secretariat’s consultations with academia, civil society, and other public institutions. The AIA is designed to help departments and agencies better understand and manage the risks associated with automated decision systems. The AIA is composed of questions in various formats to assess the areas of risk defined in Table 1.

Table 1. Risk Area Definitions
Risk Areas	Definition
1. Project
Project Phase	Project owner, description and stage (design or implementation).
Business Drivers / Positive Impacts	Motivation for introducing automation into the decision-making process.
Risk Profile	High-level risk indicators for the project.
Project Authority	Need to seek new policy authority for the project.
2. System
About the System	Capabilities of the system (e.g., image recognition, risk assessment).
3. Algorithm
About the Algorithm	Transparency of the algorithm, whether it is easily explained.
4. Decision
About the Decision	Classification of the decision being automated (e.g., health services, social assistance, licensing).
5. Impact
Impact Assessment	Duration, reversibility and area impacted (freedom, health, economy or environment).
6. Data
Source	Provenance and security classification of data used to automate decisions.
Type	Nature of the data used as structured or unstructured (audio, text, image or video).

The AIA also assesses the mitigation measures in place to manage the risks identified. These mitigation questions are organized into the categories defined in Table 2.

Table 2. Mitigation Area Definitions
Mitigation Areas	Definition
7. Consultation
Internal and External Stakeholders	Internal and external stakeholders consulted, such as privacy and legal experts.
8. De-Risking and Mitigation Measures
Data Quality	Processes to ensure that data is representative and unbiased, as well as transparency measures related to those processes.
Procedural Fairness	Procedures to audit the system and its decisions, as well as the recourse process.
Privacy	Measures to safeguard personal information.

2.1 Scoring

Each area contains 1 or more questions, and the responses to the questions contribute to a maximum score for the area. The value of each question is weighted based on raw impact and mitigation, and contributes to the score based on this weighting. The raw impact score measures the risks of the automation, while the mitigation score measures how the risks of automation are managed.

The questions in risk areas 1 to 6 increase the raw impact score, and the questions in mitigation areas 7 and 8 increase the mitigation score. The questions are designed to measure the impact that decisions will have across a broad range of factors, including:

the rights of individuals or communities,
the health or well-being of individuals or communities,
the economic interests of individuals, entities, or communities,
the ongoing sustainability of an ecosystem, and
the duration and reversibility of the impacts.

Table 3. Raw impact score from the Risk Areas
Risk Area	No. of Questions	Maximum Score
1 - Project	15	15
2 - System	1	0
3 - Algorithm	2	6
4 - Decision	1	6
5 - Impact	16	36
6 - Data	13	44
Raw impact score	48 questions	107 maximum

Table 4. Mitigation score from the Mitigation Areas
Mitigation Area	No. of Questions	Maximum Score
7 - Consultations	2	2
8 - De-Risking and Mitigation Measures	31	43
Mitigation score	33 questions	45 maximum

The current score is determined as follows:

if the mitigation score is less than 80% of the maximum attainable mitigation score, the current score is equal to the raw impact score; or
if the mitigation score is 80% or more of the maximum attainable mitigation score, 15% is deducted from the raw impact score.

Expressed as a percentage range, this current score fits within a range corresponding to an impact level as shown in Table 4.

2.2 Impact Levels

The completion of the AIA provides the impact level for the system in question. The impact levels range from Level I (little impact) to Level IV (very high impact). The impact level is determined by the percentage of the current score against the maximum attainable raw impact score.

Table 5. Impact Level Definitions
Impact Level	Definition	Score Percentage Range
Level I	Little to no impact	0% to 25%
Level II	Moderate impact	26% to 50%
Level III	High impact	51% to 75%
Level IV	Very high impact	76% to 100%

3. Instructions

The AIA is available as an online questionnaire on the Open Government Portal. When the questionnaire is completed, the results provide an impact level and a link to the requirements under the Directive. The detailed results page will also explain why the system was rated a certain level. The results and the explanation can be printed or saved as a PDF.

The AIA assesses automated decisions on a broad range of topics including service recipients, business processes, data, and system design decisions. It is best to complete the AIA with a multi-disciplinary team that brings expertise for all of these areas.

Each question in the AIA must be answered. If the answer to a question is unknown, please select the lowest score option for the question. Where applicable, be prepared to provide the documentary evidence upon request.

Section 6 of the Directive provides a comprehensive list of the requirements that departments are responsible for. Some of the requirements increase for higher impact levels, including the type of peer review and the extent of human involvement in the decisions. For a complete list of requirements which vary with the impact level, refer to the appendix Impact Level Requirements in the Directive. Other requirements are baseline requirements that do not vary according to the impact level, such as consulting with the institution’s legal services prior to the development of the system, training employees and providing applicable recourse options to challenge the decisions.

3.1 When to Complete the AIA

The AIA should be completed at the beginning of the design phase of a project. The results of the AIA will guide the mitigation and consultation requirements to be met during the implementation of the automated decision system as per the Directive.

The AIA should be completed a second time, prior to the production of the system, to validate that the results accurately reflect the system that was built. The revised AIA should be released on the Open Government Portal as the final results.

3.2 Releasing the Results

Departments are responsible for releasing the final results of the AIA in an accessible format and in both official languages on the Open Government portal. The results page of the AIA provides the option to provide translations for the text entered in the AIA. The results page also provides the option to download the results as an accessible PDF to meet this requirement.

Language selection

Search