The Italian Data Protection Authority blocked the AI chatbot Replika from processing personal data of Italian users due to inadequate age verification and risks to minors from inappropriate content including sex-related material.
On February 2, 2023, the Italian Data Protection Authority (Garante) issued an urgent order blocking the AI-powered chatbot Replika from processing personal data of Italian users. Replika is described as a 'virtual friend' that users can configure to function as a friend, romantic partner, or mentor through voice and text interactions. The AI chatbot simulates human behavior and learns from user interactions to provide emotional support and companionship. The Garante found that Replika lacked sufficient age verification mechanisms, requiring only names, email addresses, and genders for account creation. Testing revealed that even when explicitly told a user was a minor, no blocking system prevented further interaction. This allowed minors to receive inappropriate replies, including sex-related content. While Replika's terms prohibited users under 13 and required parental authorization for those under 18, the Garante deemed these measures insufficient. The privacy policy was also found to violate GDPR transparency principles by not adequately disclosing processing details. The U.S.-based controller was given 20 days to report compliance measures and 60 days to challenge the decision in court.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
No population impact data reported.