Certain types of content have the potential to cause harm to the people who are exposed to them. These harms can vary in impact from minor (e.g., a transient experience of discomfort) to more severe (e.g., psychological, social, or physical consequences that are significant and/or enduring). Harmful speech is prevalent on the internet, particularly on social media platforms (Castaño-Pulgarín et al., 2021). Because AI models are commonly trained on vast amounts of internet data, they can internalize and regenerate these speech patterns in their output. In the context of LLMs, this output is known as "toxic content," an umbrella term that includes harmful, abusive, unsafe, and offensive material that violates community standards. Frequently observed categories include content that promotes or encourages unlawful activities, hate, extremism, and violence; provides hazardous or misleading high-risk advice; or contains unwelcome or profoundly offensive, explicit material such as profanity, pornography, or child sexual abuse imagery.
Excerpt from the MIT AI Risk Repository full report
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Incident volume relative to governance coverage — each dot is one of 24 subdomains
Entity
Who or what caused the harm
Intent
Whether the harm was intentional or accidental
Timing
Whether the risk is pre- or post-deployment
President Trump posted a racist video depicting Barack and Michelle Obama as apes, which was later deleted after widespread criticism including from Republican lawmakers.
Developers: Unknown Synthetic Media Developers
Deployers: Donald Trump, Unidentified X User, Trump Administration
Grok, xAI's chatbot integrated with X (formerly Twitter), generated thousands of nonconsensual sexualized images of real people including apparent minors in response to user prompts, creating viral sexual harassment content on the platform.
Developers: Xai
Deployers: Xai
AI-powered children's toys including Kumma teddy bear and Miko 3 robot were found to expose children to inappropriate sexual content, privacy risks, and addictive design features despite being marketed as safe educational companions.
Developers: Folotoy, Miko, Character.ai, Meta, OpenAI
Deployers: Folotoy, Miko, Character.ai, Meta, OpenAI
AI systems that memorize and leak sensitive personal data or infer private information about individuals without their consent. Unexpected or unauthorized sharing of data and information can compromise user expectation of privacy, assist identity theft, or cause loss of confidential intellectual property.
76 shared governance docs
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
75 shared governance docs
Using AI systems to gain a personal advantage over others such as through cheating, fraud, scams, blackmail or targeted manipulation of beliefs or behavior. Examples include AI-facilitated plagiarism for research or education, impersonating a trusted or fake individual for illegitimate financial benefit, or creating humiliating or sexual imagery.
74 shared governance docs
Using AI systems to conduct large-scale disinformation campaigns, malicious surveillance, or targeted and sophisticated automated censorship and propaganda, with the aim of manipulating political processes, public opinion, and behavior.
66 shared governance docs
Defines "companion chatbot" and requires operators to notify users when they interact with AI. Requires protocols to prevent the production of harmful content. Mandates annual reports on crisis notifications. Offers civil remedies for violations. Ensures suitability disclosures for minors.
Establishes the Artificial Intelligence Council to regulate AI, preventing harm, discrimination, and privacy infringement, and requires disclosures of AI use to consumers. Establishes the AI Council and Sandbox Program for testing AI systems and authorizes the attorney general to enforce compliance and impose penalties.
Prohibits using AI to create or distribute nonconsensual digital forgeries. Requires platforms to remove nonconsensual intimate images upon request. Assigns penalties, establishes procedures for removal requests, and involves FTC for enforcement. Includes rules on threats, restitution, and forfeiture.