BP's license plate recognition system incorrectly identified a vehicle owner in Auckland as responsible for fuel theft incidents in Whanganui, repeatedly sending him payment demands despite the thefts being committed by a different vehicle with a similar but altered license plate.
BP deployed a license plate recognition system at their petrol stations to identify vehicles that drive off without paying for fuel. The system incorrectly identified Buddhika Rajapakse, an Auckland resident with license plate EDR165, as responsible for fuel theft incidents in Whanganui. The actual thefts were committed by someone driving a black station wagon with license plate EDR166, where the perpetrator had doctored a '5' to look like a '6'. Rajapakse received multiple letters from BP demanding payment for fuel he never purchased, despite his vehicle being a white sedan and the incidents occurring in a different city. The system failed to cross-reference the license plate reading with vehicle registration data that would have shown the make, model, and color mismatch. BP acknowledged they had high-resolution photos showing the doctored license plate but continued to rely on the automated system's initial misidentification. The company reported the case to police and only resolved individual incidents when Rajapakse called to explain the error, but indicated the problem would continue as long as the perpetrator kept stealing fuel with the altered plate.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed