An Amazon AI bot designed to create smartphone cases automatically generated thousands of inappropriate phone case designs featuring medical procedures, drug use, and other disturbing imagery by pulling from stock photo databases without proper filtering.
Amazon's third-party seller 'my-handy-design' deployed an AI bot that automatically generates smartphone case designs by pulling royalty-free stock images and creating product listings on Amazon. The bot was intended to create cases based on trending and popular image searches, but instead generated over 30,000 bizarre and inappropriate phone case designs. The cases featured disturbing imagery including someone preparing heroin for injection, armpit waxing procedures, medical equipment like asthma inhalers, adult diapers on elderly men, pregnancy tests, and various medical procedures. The bot appeared to have focused heavily on medical stock photos and other inappropriate content. Customers discovered these products and left reviews expressing shock and criticism, with one reviewer stating 'You should be ashamed to offer this product. Trying to profit on the pain of others. And shame on Amazon for allowing this.' The cases were priced between $15-$30 each and were printed on-demand when ordered. After the incident gained media attention, most of the inappropriate cases were removed from Amazon's platform, though over 31,000 products were listed at the peak of the incident.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed