Idemia's facial recognition algorithms showed significantly higher error rates for Black women compared to other demographic groups, and a 14-year-old Black girl was wrongly identified and banned from a skating rink due to facial recognition misidentification.
Idemia, a French company providing facial recognition software to police in the US, Australia, and France, developed algorithms that showed significant demographic bias in NIST testing. In July tests by the National Institute of Standards and Technology, Idemia's algorithms falsely matched Black women's faces at a rate of one in 1,000, compared to one in 10,000 for white women - a 10-fold difference in error rates. The company's software is used by US Customs and Border Protection for cruise ship passengers and by the FBI for scanning 30 million mugshots. In a specific incident, 14-year-old Lamya Robinson was wrongly identified and banned from Riverside Arena skating rink in Livonia, Michigan. The facial recognition system matched her face at 97 percent confidence to someone who had been previously banned for fighting, despite Robinson never having visited the rink before. Staff forced her to leave the premises alone, creating a safety risk. Her parents are considering legal action, with her mother calling it 'racial profiling.' The NIST study found that many facial recognition systems from over 50 companies showed similar demographic performance gaps, with white males typically having the lowest false match rates and Black females the highest.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Accuracy and effectiveness of AI decisions and actions are dependent on group membership, where decisions in AI system design and biased training data lead to unequal outcomes, reduced benefits, increased effort, and alienation of users.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed