A 13-year-old Tennessee student was arrested, strip-searched, and jailed after school surveillance software Gaggle flagged her offensive joke about 'killing all the Mexico's' as a threat, despite the comment being made in context during teasing from friends.
In August 2023, a 13-year-old girl at Fairview Middle School in Tennessee made an offensive joke while chatting with classmates on her school email system. After friends teased her about her tanned complexion and called her 'Mexican,' she wrote 'on Thursday we kill all the Mexico's.' The school's surveillance software Gaggle flagged this comment and immediately notified law enforcement. The student was arrested before the morning was over, interrogated, strip-searched, and spent the night in jail. Her parents were not allowed to speak with her until the next day. A court ordered eight weeks of house arrest, psychological evaluation, and 20 days at an alternative school. This incident occurred under Tennessee's 2023 zero-tolerance law requiring immediate law enforcement reporting of any threat of mass violence against schools. Gaggle's CEO Jeff Patterson stated the school system did not use the software as intended, which should focus on early intervention rather than immediate law enforcement involvement. The incident highlights broader concerns about AI-powered surveillance systems in schools that monitor student communications and can trigger immediate law enforcement responses without considering context.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed