The Canada Revenue Agency spent nearly $20 million on an AI chatbot called Charlie that provided accurate answers only 44% of the time when tested, giving wrong tax information to Canadians.
The Canada Revenue Agency developed an AI chatbot called Charlie the Chatbot, announced in 2020 by then-National Revenue Minister Diane Lebouthillier with a budget of over $18 million. The project cost $13.67 million in salaries, benefits and travel for staff who worked on it, plus $3.21 million for IT consultants. When tested by Auditor General Karen Hogan's team, Charlie provided accurate answers in only 2 out of 6 questions asked, achieving 44% accuracy overall. In comparison, other public web-based AI tools answered 5 out of 6 questions accurately. The chatbot is currently available on 13 different CRA webpages and has been used for over 7 million conversations with over 18 million questions since inception. The CRA reported Charlie recently reached a 70% accuracy threshold, with a new unreleased version achieving approximately 90% accuracy. However, the agency acknowledged that the exact number of correct answers cannot be precisely determined without reviewing all interaction transcripts. The CRA's 100-day service improvement plan reported that 70% of Charlie users reached their goal in November.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed