The UK government's use of AI across multiple departments, including algorithms for benefit fraud detection, facial recognition by police, and marriage license screening, has produced discriminatory results against ethnic minorities and vulnerable groups.
The UK government has deployed AI systems across at least eight Whitehall departments and police forces to make decisions on welfare, immigration, and criminal justice matters. The Department for Work and Pensions (DWP) has invested £70 million in machine learning algorithms to detect benefit fraud, with the system flagging potentially fraudulent Universal Credit claims for human review. Labour MP Kate Osamor reported that dozens of Bulgarian nationals had their benefits suspended, potentially due to the DWP's algorithm. The Home Office uses an algorithm to detect sham marriages that disproportionately flags people from Albania, Greece, Romania, and Bulgaria. Several police forces use facial recognition cameras that, when sensitivity settings are lowered, incorrectly detect at least five times more black people than white people. The DWP system has shown evidence of bias toward older claimants and has limited ability to test for unfair impacts across protected characteristics. The National Audit Office warned of 'inherent risk' that algorithms could be biased toward selecting claims from vulnerable people or groups with protected characteristics. These AI tools affect millions of people, with Universal Credit alone used by 5.9 million people as of April 2023.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Unequal treatment of individuals or groups by AI, often based on race, gender, or other sensitive characteristics, resulting in unfair outcomes and unfair representation of those groups.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed