The US Social Security Administration deployed a ChatGPT-style AI chatbot called Agency Support Companion to assist employees, but failed to warn staff not to upload sensitive personal information in the training video, while Brazil's AI-powered social security app Meu INSS systematically rejected legitimate claims from rural workers due to minor errors.
The US Social Security Administration deployed an AI chatbot called the Agency Support Companion to assist employees with everyday tasks and enhance productivity. The launch was accompanied by a poorly made training video featuring a four-fingered animated woman that failed to warn staff not to upload sensitive personal information to the system, forcing the agency to send an apologetic email highlighting this critical oversight. Employees reported the chatbot provided vague and inaccurate responses and was largely ignored by staff. Separately, Brazil's AI-powered social security app Meu INSS, launched in 2018 by state-owned company Dataprev, uses computer vision and natural language processing to analyze documents submitted by claimants. The app has systematically rejected legitimate claims from rural workers based on minor errors, including mistakenly identifying 55-year-old Josélia de Brito as a man and denying her retirement benefits. Rural workers face particular challenges due to high illiteracy rates (15% vs 5% in urban areas) and limited digital literacy. The app processes 84 million hits monthly and aims to handle 55% of all filings by 2025, but appeals take an average of 278 days to process.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed