AI Spam Filters Allegedly Block Legitimate Emails Based on Biased Keyword Detection

AlgorithmWatch conducted an experiment sending hundreds of emails to 10 email inboxes at Gmail, Yahoo, Outlook, GMX and LaPoste using accounts created specifically for the experiment. The results showed that Microsoft Outlook's spam filter marked emails as spam based on single words: an internship application from a Nigerian student was filtered when containing 'Nigeria' but delivered when the word was removed; a sex education program description was filtered with 'sex' but delivered without it; and an excerpt from a Joe Biden speech on student debt was filtered until words like 'loan', 'investment' and 'billion' were removed. Other email providers did not display this behavior. Microsoft declined to comment on the findings. The researchers determined that machine learning algorithms likely identified these words as discriminators between spam and legitimate messages, with Microsoft not making their training dataset available for review. SpamAssassin, an open-source spam filter, was also found to have similar issues, with its default configuration flagging words like 'Ivory Coast', 'Nigeria' and 'Nigerian government' as potentially spammy, and the phrase 'Oprah!' listed as potentially spammy though inactive. In SpamAssassin's 15-year-old public corpus still widely used for training, 59 out of 1,397 spam emails were from Nigerians while none were in the legitimate email folder.