Risks to privacy

International Scientific Report on the Safety of Advanced AI

Bengio et al. (2024)

Source

Sub-category

Risk Domain

2Privacy & Security

2.1Compromise of privacy by leaking or correctly inferring sensitive information

AI systems that memorize and leak sensitive personal data or infer private information about individuals without their consent. Unexpected or unauthorized sharing of data and information can compromise user expectation of privacy, assist identity theft, or cause loss of confidential intellectual property.

"General- purpose AI models or systems can ‘leak’ information about individuals whose data was used in training. For future models trained on sensitive personal data like health or financial data, this may lead to particularly serious privacy leaks. General- purpose AI models could enhance privacy abuse. For instance, Large Language Models might facilitate more efficient and effective search for sensitive data (for example, on internet text or in breached data leaks), and also enable users to infer sensitive information about individuals."(p. 60)

Entity— Who or what caused the harm

Human

Due to a decision or action made by humans

AI system

Due to a decision or action made by an AI system

Other

Due to some other reason or is ambiguous

Intent— Whether the harm was intentional or accidental

Intentional

Due to an expected outcome from pursuing a goal

Unintentional

Due to an unexpected outcome from pursuing a goal

Other

Without clearly specifying the intentionality

Timing— Whether the risk is pre- or post-deployment

Pre-deployment

Occurring before the AI is deployed

Post-deployment

Occurring after the AI model has been trained and deployed

Other

Without a clearly specified time of occurrence

Supporting Evidence (4)

"General- purpose AI systems rely on and process vast amounts of personal data, and this could pose significant and potentially wide- reaching privacy risks. Such risks include loss of data confidentiality for people whose data was used to train these systems, loss of transparency and control over how data- driven decisions are made, and new forms of abuse that these systems could enable."(p. 60)

"Many systems are trained on publicly available data containing personal information without the knowledge or consent of the individuals it pertains to. This information could then be outputted by a general- purpose AI system in undesired contexts. There is a risk that training models on sensitive data containing personal information (such as medical or financial data) could result in serious privacy leaks. It is difficult to assess the likelihood or potential impact of these risks: for example, existing medical general- purpose AI systems such as Google’s Gemini- Med (596*) are only trained on anonymised public patient data, and the rate at which such models regurgitate training data has not yet been studied. General- purpose AI systems that continuously learn from interactions with users (e.g. chatbots such as ChatGPT) might also leak such interactions to other users, although at the time of writing, there are no well- documented cases of this occurring."(p. 61)

"Privacy is a challenging concept to define (583). In the context of AI it encompasses: • Data confidentiality and protection of personal data collected or used for training purposes, or during inference (584) • Transparency, and controls over how personal information is used in AI systems (585), for example the ability for individuals to opt- out from personal data being collected for training, or the post- hoc ability to make a general- purpose AI system ‘unlearn’ specific information about an individual (586); • Individual and collective harms that may occur as a result of data use or malicious use, for example the creation of deepfakes (587)."(p. 61)

"General- purpose AI systems could enable privacy abuse. Some studies have found that general- purpose AI systems have privacy- relevant capabilities that may be exploited by malicious users of these systems. For example, fine- grained internet- wide search capabilities, such as powerful reverse image search or forms of writing style detection, which allow individuals to be identified and tracked across online platforms, or sensitive personal characteristics to be inferred, further eroding individual privacy (597, 598). Large language models could also enable more efficient and effective search for sensitive information on the internet, or in breached datasets. General- purpose AI- generated content, such as non- consensual deepfakes, could be used to manipulate or harm individuals, raising concerns about the harm caused by the malicious use of personal data and the erosion of trust in online content (255, 256, 373, 599)."(p. 61)