2.1 Compromise of privacy by leaking or correctly inferring sensitive information

In the context of generative AI, privacy violations arise when systems collect and divulge sensitive information that individuals or corporations do not consent to sharing with others. Privacy violations can occur both accidentally and intentionally.

Examples of accidental causes include AI models that memorize and inadvertently reproduce or leak sensitive personal information present in their training data, such as names, addresses, and medical records. Even when personal data is not included in the training dataset or directly offered by the user, models can make inferences about sensitive or protected traits of individuals based on predictive correlations within their history of interactions, build profiles of users, or train AI systems. As a result, models may save and reproduce sensitive information derived from prior interactions, such as classified intellectual property. A notable example is the case where Samsung employees accidentally leaked confidential intellectual property to OpenAI after using ChatGPT to help with coding tasks.

Intentional causes include the malicious design and use of AI to exploit users' trust by influencing them to share personal or private information about themselves or others. Privacy attacks, such as membership inference, could allow adversaries to gain knowledge of the private records used to train an AI model. Malicious actors could also deliberately extract private information from a model by crafting prompts designed to exploit the model's knowledge of sensitive data.

Excerpt from the MIT AI Risk Repository full report