Facebook's AI content moderation system incorrectly flagged the word 'hoe' (a gardening tool) as violating community standards in a gardening group, threatening the group with deletion and disrupting legitimate gardening discussions.
Facebook's AI-powered content moderation system repeatedly flagged posts containing the word 'hoe' in the WNY Gardeners Facebook group, a gardening community with over 7,500 members in western New York. The AI system incorrectly identified references to the gardening tool as violations of community standards for harassment and bullying, apparently confusing it with a different word. When a group member posted 'Push pull hoe!' in response to a question about weeding tools, Facebook sent a notification stating the comment violated standards for harassment and bullying. The group's moderator, Elizabeth Licata, was concerned that repeated violations could lead to the entire group being shut down. She attempted to contact Facebook to explain that 'hoe' referred to a gardening tool in their context, but found the company's support system ineffective and never received a response from an actual person. The AI system also flagged other gardening-related comments about killing beetles with soapy water as 'possible violence, incitement, or hate.' After being contacted by The Associated Press, Facebook acknowledged the errors and implemented additional human review for the group to prevent future automated enforcement mistakes.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed