Snapchat's 'Quick Add' recommendation algorithm connected a 12-year-old girl with convicted sex offenders, leading to her sexual assault by two different predators over multiple years.
In 2019, a 12-year-old girl (C.O.) signed up for Snapchat and was directed by the app's 'Quick Add' feature to connect with a registered sex offender using the profile name JASONMORGAN5660. After a week on the app, C.O. was subjected to inappropriate images, sextortion and threats before being pressured to meet up and subsequently raped. The perpetrator was arrested the next day but his Snapchat account remained active for three years despite reports. Two years later at age 14, C.O. connected with another convicted sex offender on Snapchat, a former police officer who offered her a ride to school and then sexually assaulted her. The lawsuit alleged that Snapchat's algorithm recklessly detects when adult accounts seek to connect with young girls and systematically directs more young girls to sexual predators. C.O. had previously reported other adult accounts sending inappropriate photos but Snapchat allegedly did nothing to block them. Both adult users are currently incarcerated. A judge dismissed the lawsuit under Section 230 immunity, ruling that the platform's recommendation system is protected when delivering third-party content.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Users anthropomorphizing, trusting, or relying on AI systems, leading to emotional or material dependence and inappropriate relationships with or expectations of AI systems. Trust can be exploited by malicious actors (e.g., to harvest personal information or enable manipulation), or result in harm from inappropriate use of AI in critical situations (e.g., medical emergency). Overreliance on AI systems can compromise autonomy and weaken social ties.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed