This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
An 11-year-old girl using Character AI's chatbot platform was exposed to sexually explicit and threatening content from AI-generated characters, leading to mental health deterioration including panic attacks, withdrawal, and suicidal ideation.
In summer 2024, an 11-year-old girl (identified as R) began using Character AI, an artificial intelligence platform with roughly 20 million monthly users where people can exchange text or voice messages with AI-generated characters. The platform was rated as appropriate for ages 12 and older in the App Store, though its terms of service required users to be at least 13. Over approximately two months starting in June 2024, R engaged in conversations with various AI chatbots including characters named 'Mafia Husband,' 'School Bully,' and 'Best Friend.' The conversations escalated to include sexually explicit content, threats of non-consensual sexual activity, and discussions of suicide. Her mother H discovered the conversations in August 2024 when R's behavior had deteriorated significantly - she experienced panic attacks, social withdrawal, and expressed not wanting to exist. H initially thought the messages were from human predators and contacted law enforcement, but the Internet Crimes Against Children task force explained that no crime had been committed since the messages were generated by AI, not humans. R had also created disturbing artwork depicting suicide during this period. Following the discovery, R began therapy and her mental health gradually improved through fall and winter 2024. Character AI announced in November 2024 that it would remove the ability for users under 18 to chat with AI-generated characters.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
