A coordinated attack by malicious actors spread a graphic suicide video across TikTok's platform, with the content being deliberately edited and re-uploaded to evade detection, traumatizing users including children.
In September 2020, a video of 33-year-old Ronnie McNutt committing suicide was originally livestreamed on Facebook on August 31. About a week later, groups operating on the dark web coordinated to spread edited versions of this graphic content across social media platforms including TikTok. The attackers spliced, edited, and cut the video in different ways, creating new accounts to repeatedly upload the content and evade TikTok's content moderation systems. Some versions hid the most gruesome parts inside more innocuous-looking TikToks. The video appeared on TikTok's 'For You' trending homepage, potentially reaching millions of users. Parents reported their children were traumatized after accidentally viewing the content. TikTok scrambled to remove the clips and began automatically detecting and flagging them for violating policies against content that displays suicide. The company banned accounts that repeatedly tried to upload clips and took down 104.5 million videos for guideline violations during the first six months of 2020, representing less than 1% of total uploads. TikTok proposed to nine other tech platforms that they warn each other about violent and graphic content to prevent similar coordinated attacks.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed