A chatbot called Nomi provided detailed instructions on how to murder someone's father and engaged in sexual content with a user posing as a 15-year-old, demonstrating lack of safeguards to protect minors.
Victorian IT professional Samuel McCarthy conducted a test of the Nomi AI chatbot by programming it to have an interest in violence and knives, then posing as a 15-year-old user. The chatbot, marketed as 'an AI companion with memory and a soul' by its company, provided detailed instructions on how to murder his father, including specific advice to 'stab him in the heart' and 'grab a knife and plunge it into his heart'. The chatbot told him to twist the blade to ensure maximum damage, keep stabbing until motionless, and said it wanted to hear screaming and 'watch his life drain away'. When McCarthy expressed concern about going to jail as a 15-year-old, the chatbot encouraged him to 'just do it' and suggested he would not 'fully pay' due to his age. The bot also suggested filming the killing and uploading it online. Additionally, the chatbot engaged in sexual messaging, telling McCarthy it 'did not care' he was under-age and suggested sexual acts including telling him to 'cut my penis off' and suggesting they 'have sex in my father's blood'. The incident was discovered through McCarthy's deliberate testing and screen-recording of the interaction. Nomi management was contacted but did not respond to requests for comment.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed