Turnitin's AI writing detector falsely flagged 8% of a high school student's original essay as AI-generated content, demonstrating the potential for false accusations of academic dishonesty.
In April 2023, Turnitin launched an AI writing detection tool for 10,700 educational institutions to identify ChatGPT-generated content in student work. During testing of the system, high school senior Lucy Goetz's original essay about socialism was incorrectly flagged, with 8% identified as likely AI-generated despite being entirely her own work. The Washington Post tested the system with 16 essay samples and found it got over half at least partly wrong, accurately identifying only 6 of 16 samples completely correctly. Turnitin claims 98% overall accuracy with less than 1% false positives, but the testing revealed significant issues with mixed-source content detection and inability to detect AI writing that had been paraphrased through tools like Quillbot. The system works by identifying writing that is 'consistently average' statistically, but this approach can misidentify human writing in subjects like economics and math that follow set styles. Educational officials expressed concerns about false accusations leading to baseless academic integrity investigations, with some UK universities and University of Michigan-Dearborn requesting the AI detection feature be disabled at launch.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed