Guide on the use of generative AI

Issue: some generative AI tools do not meet government information security requirements

The protection of personal, classified, protected and proprietary information is critical when using generative AI systems. The suppliers of some generative AI tools may inspect input data or use it to further train their models, which could result in privacy and security breaches. Risks can also arise from input data being stored on servers not controlled by the GC, where data might be retained for longer than necessary, made accessible, further distributed, or vulnerable to a data breach. ^{Footnote 16} Some tools, public or otherwise, may not meet the privacy and security requirements set out in federal law and policy. Developing or deploying generative AI systems on institutions’ secure networks may mitigate some risks, and institutions should seek approval from their internal officials, such as their chief security officer, designated official for cyber security and chief information officer. As part of these approvals, a formal risk assessment and acceptance is recommended because additional mitigation measures may be needed to lower risks to an acceptable level.

Best practices for all users of generative AI in federal institutions

• Don’t enter sensitive or personal information into any tools not managed by the GC.
• Don’t submit queries on non-GC managed tools that could undermine public trust if they were disclosed. See Appendix B of the Directive on Service and Digital for examples of unacceptable network and device uses.
• Understand how a system uses input data (for example, whether it’s used as training data and whether it’s accessible to suppliers).
• Ask legal services, the institutional chief security officer, and the privacy team to review a supplier’s terms of use, privacy policy and other legal documents before using any system to process sensitive or proprietary information.
• Use infrastructure and tools that are appropriate for the security classification of the information, in accordance with the Directive on Security Management.
• Seek the approval of the institutional chief security officer before using, procuring or deploying generative AI for protected or other sensitive information.
• Consider the requirements for information and data residency in the Directive on Service and Digital and related guidance in the Guideline on Service and Digital.
• Use the “opt-out” feature, where possible, to ensure that prompts are not used to train or further develop an AI system.

Additional best practices for federal institutions deploying a generative AI tool

• Conduct regular system testing before and during the operation of a system to ensure that risks of potential adverse impacts, as well as risks associated with inappropriate or malicious use of the system, are identified and mitigated.
• Apply more in-depth testing methods to identify and mitigate vulnerabilities in instances where systems will be made publicly available. This should include penetration testing, adversarial testing or red teaming.
• Plan independent audits for assessing generative AI systems against risk and impact frameworks. Leverage existing risk management frameworks, when appropriate. See the guidance on the Risk management page.
• Develop a plan to document and respond to cyber security events and incidents, aligned with the Government of Canada Cyber Security Event Management Plan (GC CSEMP). Update the plan as needed to address the evolving threat environment.
• Consider best practices such as the United Kingdom’s Guidelines for Secure AI System Development (available in English only).

Issue: generated content may amplify biases or other harmful ideas that are dominant in the training data

Generative AI tools can produce content that is discriminatory or not representative, or that includes biases or stereotypes (for example, biases relating to multiple and intersecting identity factors such as gender, race and ethnicity). ^{Footnote 17 Footnote 18 Footnote 19} Many generative models are trained on large amounts of data from the Internet, which is often the source of these biases. For example, training data is likely to reflect predominant historical biases and may not include perspectives that are less prevalent in the data or that have emerged since the model was trained. ^{Footnote 17} Other sources that may contribute to biased content include data filtering, which can amplify the biases in the original training set, ^{Footnote 20} framing of the prompt, ^{Footnote 21} and model bias. Widespread use of these technologies could amplify or reinforce these biases and dominant viewpoints, and lead to less diversity in ideas, perspectives and language, ^{Footnote 17 Footnote 22} as well as potential harms.

Best practices for all users of generative AI in federal institutions

• Learn about bias, diversity, inclusion, anti-racism, and values and ethics to improve your ability to identify biased, non-inclusive or discriminatory content.
• Review generated content to make sure it aligns with GC commitments, values and ethics, and meets legal obligations. This review includes assessing for biases and stereotypical associations.
• Formulate prompts to generate content that provides holistic perspectives and minimizes biases.
• Strive to understand the data that was used to train the tool, for example, where it came from, what it includes, and how it was selected and prepared.
• Clearly indicate when content has been produced by generative AI.

Additional best practices for federal institutions deploying a generative AI tool

• Consider potential biases and approaches to mitigating them from the planning and design stage, including using gender-based analysis plus (GBA Plus) to understand how deploying a generative AI tool might impact different population groups. Don’t deploy the tool if you can’t manage the risk of biased outputs.
• Consult GBA Plus experts in your organization and consult people who would be directly impacted by the deployment of the tool (such as clients) in the planning and design stages, as well as in the evaluation and auditing stages, to identify impacts of the use of these tools on different population groups and to develop measures to address any negative impacts.
• Test for biases in the data, model and outputs before deploying a tool, and on an ongoing basis.
• Regularly monitor the system for adverse impacts after deployment.
• Reflect population diversity when deploying virtual assistants, for example, by varying the gender of the assistant.

Issue: generated content may be inaccurate, incoherent or incomplete

Generative AI technologies can produce content that appears to be well developed, credible and reasonable but that is in fact inaccurate, nonsensical or inconsistent with source data. ^{Footnote 23 Footnote 6} This content is sometimes referred to as a “hallucination.” Also, content generated by AI tools may not provide a holistic view of an issue. Instead, it may focus on prevalent perspectives in the training data. ^{Footnote 17} Content might be out of date, depending on the time period the training data covers and whether the system has live access to recent data. There may also be differences in the quality of the outputs in English and French, depending on the model, task and prompt. The output quality in each language should be evaluated to ensure compliance with official language requirements.

The risks associated with inaccurate content will vary depending on the context and should be assessed. For example, using generative AI tools to learn about a topic may produce incorrect information or non-existent sources. ^{Footnote 24} If the results were to be used in decision-making, they could lead to unfair treatment of some people or to misguided policy. As well, when considering the use of these tools in public-facing communications, it is critical that the government not share inaccurate information, which would contribute to misinformation and erode public trust.

Best practices for all users of generative AI in federal institutions

• Clearly indicate that you have used generative AI to develop content.
• Don’t consider generated content as authoritative. Review it for factual and contextual accuracy by, for example, checking it against information from trusted sources or by having a colleague with expertise review the response.
• Verify personal information created using generative AI to make sure it is accurate, up to date and complete.
• Assess the impact of inaccurate outputs. Don’t use generative AI when factual accuracy or data integrity is needed.
• Strive to understand the quality and source of training data.
• Consider your ability to identify inaccurate content before you use generative AI. If you can’t confirm the quality of the content, don’t use it.
• Learn how to create effective prompts and provide feedback to refine outputs to minimize the generation of inaccurate content.
• Don’t use generative AI tools as search engines unless sources are provided so that you can verify the content.

Additional best practices for federal institutions deploying a generative AI tool

• Test performance and robustness across a variety of uses before deployment. This includes making sure the quality of tools and outputs meets official languages requirements.
• Assess training data quality when refining models.
• Use grounding and prompt engineering so the models build responses from only the information you provide and control.
• Notify users that they are interacting with generative AI.
• When content is generated by AI, include links to authoritative sources to provide users with additional context and foster transparency.
• Provide information about the appropriate use of the tools, capabilities and limitations of the system, risk mitigation measures, source of training data and how models were developed.
• Monitor the performance of generative AI tools on an ongoing basis to understand potential impacts and make sure they are meeting performance targets. Document problems, pause deployment, and update the tool if performance levels are not being achieved.

Issue: overreliance on AI could unduly interfere with judgment, stifle creativity and erode workforce capabilities

Overreliance on generative AI tools could interfere with individual autonomy and judgment. For example, some users may be prone to uncritically accept system recommendations or other outputs, which could be incorrect. ^{Footnote 25 Footnote 26} Overreliance on AI systems can be a sign of automation bias, which is the tendency to favour results generated by automated systems, even in the presence of contrary information from non-automated sources. ^{Footnote 25} As well, confirmation bias could contribute to overreliance ^{Footnote 25} because the outputs of generative AI systems can reinforce a user’s preconceptions, especially when prompts are written in a way that reflects their assumptions and beliefs. ^{Footnote 27} Excessive dependence on AI systems may result in a decline in critical thinking. This could limit diversity in thought, stifle creativity and innovation and lead to partial or incomplete analyses. As such, overreliance on AI may impede employees’ abilities to build and maintain the skills they need to perform tasks that are assigned to generative AI systems, which could potentially erode workforce capabilities.

Best practices for all users of generative AI in federal institutions

• Consider whether you need to use generative AI to meet user and organizational needs.
• Consider the abilities and limits of generative AI when assigning tasks and reviewing system outputs.
• Build your AI literacy so that you can critically assess these tools and their outputs.
• Use generative AI tools as aids, not as substitutes. Do not outsource a skill that you do not understand or possess.
• Form your own views before seeking ideas or recommendations from AI tools.
• Use neutral wording when formulating prompts to minimize biased outputs.
• Always review content generated by AI, even if the system seems to be reliable in providing accurate responses.

Issue: generative AI poses risks to human rights, privacy, intellectual property protection, and procedural fairness

The government’s use of generative AI systems poses risks to the legal rights and obligations of federal institutions and their clients. These risks arise from the data used to train AI models, the way systems process input data, and the quality of system outputs.

The use of copyright-protected materials like articles, books, code, paintings or music by suppliers or federal institutions to train AI models may infringe on intellectual property rights. The use or reproduction of the outputs generated by these models could also infringe on such rights if they contain material that is identical or substantially similar to a copyright-protected work. Further, the ownership of content created by or with the help of generative AI is uncertain. Privacy rights could also be at risk because data used to train generative AI models could include unlawfully collected or used personal information, such as personal information obtained from publicly accessible online sources.

Risks could also arise from the opacity of generative AI models and their potential for producing inaccurate, biased or inconsistent outputs. This opacity makes it difficult to trace and understand how the AI system produces outputs, which can undermine procedural fairness in instances where a federal institution is obliged to provide clients with reasons for administrative decisions, such as decisions to deny benefits. The quality of AI outputs can also impact individuals’ legal rights. For example, biased outputs could lead to discrimination in services, potentially violating human rights.

These risks extend beyond decision-making scenarios. When federal institutions use generative AI tools to help the public find information (for example, chatbots on institutional websites) or to produce public communications, there’s a risk that these tools will generate inappropriate content or misinformation that could contribute to or cause harm for which the government could be liable.

Best practices for all users of generative AI in federal institutions

• Consult your institution’s legal services about the legal risks of deploying generative AI tools or using them in service delivery. The consultation could involve a review of the supplier’s terms of use, copyright policy, privacy policy and other legal documents.
• Comply with the Directive on Automated Decision-Making when using generative AI in administrative decision-making.
• Check whether system outputs are identical or substantially similar to copyright-protected material (look at the FAQs for more information on how to do this). Give proper attribution, where appropriate, or remove the problematic material to minimize the risk of intellectual property infringement.
• Consult designated officials on the licensing and administration of Crown copyright if you are planning to include outputs in public communications, in accordance with the Procedures for Publishing.
• Evaluate system outputs for factual inaccuracies, biases or harmful ideas that may conflict with GC values.
• Keep up to date on legal and policy developments related to AI regulation.

Additional best practices for federal institutions deploying a generative AI tool

• Verify the legality of the method used to obtain data for training AI models and make sure you have permission to use the data for this purpose.
• Document the data provenance and that you have authorization from the copyright owner.
• Where feasible, train your model using open-source data that you have confirmed you may use in this way.
• Be transparent when you use generative AI. For example, notify users if they are interacting with a system rather than a human. Where relevant, include a disclaimer to minimize liability risks and increase transparency.

Issue: people may not know that they are interacting with an AI system, or they may wrongly assume that AI is being used

Conversational agents or chatbots that use generative AI can produce responses that are so human-like that it may be difficult to distinguish them from those of a real person. ^{Footnote 28} As a result, clients may think they are interacting with a human. Similarly, clients may think an email they have received was written by a person when it was actually generated by an AI tool. On the other hand, clients might think they are interacting with an AI tool when they are actually dealing with a real person. Transparency about whether a client is interacting with a person or a chatbot is essential to ensure that the client is not misled and to maintain trust in government.

Best practices for all users of generative AI in federal institutions

• Clearly communicate when and how you are using AI in interactions with the public.
• Inform users when text, audio or visual messages addressed to them are generated by AI.

Additional best practices for federal institutions deploying a generative AI tool

• Offer alternative non-automated means of communicating.
• Use watermarks to identify content that is generated by AI.
• Publish information about the system, such as a plain-language description of how it works, why your institution is using it and what steps were taken to ensure the quality of the outputs.

Issue: the development and use of generative AI systems can have significant environmental costs

The development and use of generative AI systems can be a significant source of greenhouse gas (GHG) emissions and water usage. These emissions come not only from the compute used to train and operate generative models but also from the production and transportation of servers that support AI programs. In addition, data centres are energy-intensive and consume vast quantities of water for on-site cooling and off-site electricity generation. ^{Footnote 29} Although generative AI has the potential to help combat climate change, its use must be balanced against the need for swift and drastic action to reduce global greenhouse gas emissions and avert irreversible damage to the environment. ^{Footnote 30}

Best practices for all users of generative AI in federal institutions

• Use generative AI tools hosted in net-zero or carbon-neutral data centres.
• Use generative AI tools only when relevant to program objectives and desired outcomes.
• Understand that the environmental impact of generative AI comes not only from its use but also from the training of the models and the manufacturing of the hardware components that run data centres, as well as the mining, manufacturing, assembly, transportation, and disposal of them.

Additional best practices for federal institutions deploying a generative AI tool

• Find out whether your AI supplier has set any greenhouse-gas reduction targets. ^{Footnote 31}
• Conduct an environmental impact assessment as part of the proposal to develop or procure generative AI tools. Make sure any decision to procure these tools is made in accordance with the Policy on Green Procurement.
• Encourage developers to be transparent about sustainability by giving preference to those that clearly communicate the environmental impacts of their AI systems through reports on compliance with GHG Protocols. ^{Footnote 32}
• Use carbon-awareness tools to estimate the carbon footprint of systems before training or production. ^{Footnote 33}