Apple's AI-powered Apple Intelligence feature prioritized and summarized phishing and scam emails as legitimate priority messages for iPhone and Mac users, making fraudulent content appear more credible.
In late 2024, Apple rolled out its Apple Intelligence AI update to millions of iPhone, iPad, and Mac users in Australia and other regions. The flagship features include notification summarizing and prioritization capabilities that use AI to condense multiple notifications and flag certain alerts as priority. However, users reported that the system was incorrectly prioritizing phishing and scam emails, making them appear more legitimate. Examples included fake tax office emails demanding payment, fraudulent Xfinity billing suspension notices, and fake USPS delivery messages. The AI system was summarizing these scam messages and marking them as priority, potentially increasing users' likelihood of falling for fraud. Screenshots shared on social media platforms like Reddit and Bluesky showed the system highlighting obvious scam emails with urgent-sounding summaries. Experts warned that this could lead to increased financial losses for users, particularly given that Australians reported losing $2.7 billion to scams in 2023. The issue appears to stem from the AI focusing on urgent language rather than checking sender authenticity or other fraud indicators. Apple told the BBC it would update the feature to clarify when text is AI-summarized, but the company did not respond to direct requests for comment about the scam prioritization issue.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed