Toronto deployed an AI predictive model to forecast water quality at two beaches, but the system incorrectly declared beaches safe for swimming on nearly 50 occasions when E. coli levels were dangerously high, exposing beachgoers to potential illness.
In June 2022, Toronto Public Health quietly implemented an artificial intelligence predictive modeling (AIPM) system developed by Montreal-based startup Cann Forecast to forecast water quality at Sunnyside and Marie Curtis beaches. The city paid CA$30,000 for this pilot program as an alternative to traditional 24-hour laboratory testing for E. coli bacteria. The AI model used historical data, rainfall, temperature, wind direction, and real-time meteorological data to predict beach safety. However, analysis by water advocacy group Swim Drink Fish found the model was incorrect about half the time, with 30 instances where swimming status was incorrectly predicted. The Information's analysis revealed the model correctly identified unsafe conditions only 38% of the time at Sunnyside Beach, leading to 31 days when the beach remained open despite dangerous E. coli levels. In one instance, Marie Curtis beach was kept open despite E. coli counts nearly 4.5 times the acceptable level. The model's predictions superseded daily testing data even when laboratory results showed high bacterial contamination. City officials defended the system despite its poor performance, and it was later discovered that faulty weather data contributed to prediction errors. The pilot program was scheduled for evaluation after Labor Day 2022.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed