Australia's Centrelink automated debt recovery system (robo-debt) incorrectly identified welfare debts by averaging annual income data, sending threatening debt notices to thousands of recipients who did not actually owe money.
In July 2016, Australia's Department of Human Services implemented an automated debt recovery system for Centrelink welfare payments, removing human oversight from the debt identification process. The system compared annual income data from the Australian Tax Office with fortnightly income reported to Centrelink, using a flawed 'income averaging' method that assumed recipients worked consistently throughout the year. This resulted in approximately 220,000 letters being sent demanding repayment of alleged overpayments, with many recipients receiving debt notices for thousands of dollars they did not actually owe. The system generated 20,000 compliance letters per week compared to 20,000 per year under the previous manual system. At least 20,000 debts were later found to be incorrect and were either reduced or wiped entirely. Recipients faced aggressive debt collection practices, travel bans, and threats of legal action. Many people reported severe psychological distress, with some becoming suicidal. The Commonwealth Ombudsman found the system lacked transparency, placed unreasonable burdens on welfare recipients, and had significant implementation flaws including poor communication, inadequate staff training, and insufficient testing before deployment.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed