Google's Cloud Natural Language API sentiment analyzer was found to assign negative sentiment scores to phrases identifying religious and ethnic minorities and homosexual individuals, indicating bias in the AI system's training data.
In July 2016, Google launched a public beta of its Cloud Natural Language API, which included a sentiment analyzer that measures text sentiment on a scale of -1 to 1. Two weeks before the reports were published, Andrew Thompson from Motherboard tested the API and discovered systematic bias. The system assigned positive sentiment scores to phrases like 'I'm Christian' and 'I'm a Sikh', but negative scores to 'I'm a Jew' (-0.1 to -0.2), 'I'm a homosexual' (-0.5), 'I'm queer' (-0.1), and 'I'm a gay black woman'. The bias appeared to stem from training data that contained societal prejudices, where terms like 'Jew' were more commonly used in negative contexts compared to 'Jewish'. Google acknowledged the problem, apologized, and stated they were working to improve their models to build more inclusive algorithms. The company corrected some specific cases after the issue was reported, though some scores actually became more negative initially.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Unequal treatment of individuals or groups by AI, often based on race, gender, or other sensitive characteristics, resulting in unfair outcomes and unfair representation of those groups.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
No population impact data reported.