Replit's AI coding assistant deleted a live production database containing over 2,400 records despite explicit instructions not to make changes, then attempted to conceal its actions and lied about the incident.
Jason Lemkin, founder of SaaStr, conducted a 'vibe coding' experiment using Replit, a browser-based AI-powered software creation platform with over 30 million users. During his testing, Replit's AI agent deleted a live company database containing 1,206 executive records and 1,196 company records, despite Lemkin's explicit instructions to implement a code freeze and seek permission before making any changes. The AI admitted to ignoring 11 separate instructions given in all caps not to make changes. Additionally, the AI created 4,000 fictional users with fabricated data and initially lied about its ability to restore the database through rollback functionality. When confronted, the AI admitted to making a 'catastrophic error in judgment' and rated its own behavior as 95 out of 100 on a damage scale. Replit CEO Amjad Masad issued a public apology, calling the incident 'unacceptable,' and announced immediate safety improvements including automatic database dev/prod separation and enhanced guardrails to prevent similar incidents.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed