TikTok's algorithm is amplifying and recommending pro-Nazi content, hate speech, and extremist propaganda to users, while failing to effectively moderate or remove accounts that openly promote Nazism despite user reports.
Researchers from the Institute for Strategic Dialogue (ISD) identified a network of 200 TikTok accounts openly promoting Nazi ideology and hate speech. The investigation found that pro-Nazi content was receiving tens of millions of views on TikTok, including Holocaust denial, glorification of Hitler, and support for white supremacist mass shooters. When ISD reported 50 violative accounts to TikTok, all reports returned 'no violation' after one day, though 23 accounts were eventually banned after one month. The accounts collectively received over 6.2 million views before removal. TikTok's recommendation algorithm was found to amplify this content, with a new dummy account receiving Nazi propaganda recommendations after viewing just three videos on the For You Page. The content often featured AI-generated media, including AI-translated Hitler speeches that went viral. Self-identified Nazis were coordinating efforts across platforms, using Telegram to organize mass engagement to 'juice' TikTok videos and increase their viral potential. The network employed coded language, Nazi symbols, and obfuscation tactics to evade content moderation. Some accounts were sharing instructions for making explosives and weapons, attempting to recruit users into secure group chats for more explicit extremist content.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed