Spain's VioGén algorithm incorrectly classified domestic violence victims as low risk, leading to inadequate protection and multiple fatalities including Lobna Hemid who was killed by her husband seven weeks after being assessed as low risk.
Spain deployed VioGén, a risk assessment algorithm used nationwide to evaluate domestic violence cases and determine protection levels for victims. The system processes answers to 35 yes/no questions about domestic violence incidents to generate risk scores from negligible to extreme risk. Since 2007, Spain has 92,000 active cases evaluated by VioGén, with 83% classified as facing little risk. However, 8% of women classified as negligible risk and 14% at low risk have reported being harmed again. At least 247 women have been killed by current or former partners since 2007 after being assessed by VioGén. In judicial reviews of 98 homicides, 55 women had been classified as negligible or low risk. Police accept VioGén's risk scores about 95% of the time. The case of Lobna Hemid illustrates the system's failures - she was beaten by her husband in January 2022, assessed as low risk by VioGén, received minimal protection, and was fatally stabbed seven weeks later. Similar cases include Stefany González Escarraman (negligible risk, killed one month later) and Eva Jaular (low risk, killed along with her daughter).
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed