Gaggle Safety Management, an AI-powered monitoring system deployed in Lawrence High School, incorrectly flagged student content including art portfolios and fitness-related messages, while also blocking a student's records request about the system itself.
In August 2023, Lawrence Public Schools in Kansas implemented Gaggle Safety Management, an AI-powered digital safety tool developed by Illinois-based Gaggle, under a three-year contract worth approximately $160,000. The system monitors all student activity on school Google and Microsoft accounts, including emails, documents, links, and calendar entries, scanning for signs of unsafe behavior such as substance abuse, violence threats, or self-harm. Between November 2023 and September 2024, Gaggle flagged over 1,200 instances of 'questionable' student content across the district, with 18 incidents reported to law enforcement and around 800 classified as 'nonissues.' The system made several notable errors, including deleting a photo of girls in tank tops from an art student's portfolio after misidentifying it as child pornography, questioning a student who wrote 'gonna die' about running in Crocs shoes, and blocking a student journalist's records request about the system itself. Students filed a lawsuit in August 2024 alleging the surveillance is unconstitutional and creates a chilling effect on their communications. The system operates continuously without opt-out options for students using school devices and accounts.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that memorize and leak sensitive personal data or infer private information about individuals without their consent. Unexpected or unauthorized sharing of data and information can compromise user expectation of privacy, assist identity theft, or cause loss of confidential intellectual property.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed