Instagram's algorithm selected a user's post containing violent rape and death threats to advertise the platform to her friends on Facebook, exposing them to the abusive content.
Instagram, owned by Facebook, used an automated algorithm to select content for advertising its platform to users' Facebook friends. The system chose a screenshot posted by Guardian reporter Olivia Solon nearly a year earlier that contained violent threats she had received via email, including text stating 'I will rape you before I kill you, you filthy whore!' and 'I Will Rape You'. The algorithm selected this post, likely because it had received engagement through likes and sympathetic comments, and used it to create an advertisement shown to Solon's sister on Facebook with the message 'See Olivia Solon's photo and posts from friends on Instagram'. Instagram apologized and stated the image was part of an effort to encourage engagement by showing content from friends to motivate people to visit the platform. The company did not answer questions about how widely the post was shared but confirmed it would have surfaced to some of Solon's Facebook friends. This incident occurred during a period when Facebook was facing scrutiny over algorithmic failures, including allowing advertisers to target antisemitic categories.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed