Google’s Automated Child Abuse Detection Wrongfully Flagged a Parent’s Naked Photo of His Child

In February 2021, two fathers - Mark in San Francisco and Cassio in Houston - took photos of their toddlers' genital infections at the request of healthcare providers for telemedicine consultations during the pandemic. Both used Android phones that automatically backed up photos to Google Photos. Google's AI system for detecting child sexual abuse material flagged these medical images, resulting in immediate account suspensions for 'harmful content' violations. The fathers lost access to their Gmail accounts, Google Photos, contacts, and phone numbers through Google Fi. Google reported both cases to the National Center for Missing and Exploited Children, which forwarded them to local police. The San Francisco and Houston police departments investigated both men and determined no crime had occurred, clearing them of any wrongdoing. However, Google refused to reinstate their accounts despite the police determinations, leaving both men permanently locked out of over a decade of digital data. Google filed over 600,000 reports of child abuse material in 2021 and disabled over 270,000 user accounts. The company's human content moderators reviewed the AI-flagged images but upheld the decisions, and Google's appeals process provided no mechanism for correction even with police clearance documentation.