Lawyers in South African courts used AI tool 'Legal Genius' to draft legal submissions that contained fictitious case citations, leading to professional misconduct investigations by the Legal Practice Council.
In the Johannesburg High Court case Northbound Processing v South African Diamond and Precious Metals Regulator, junior counsel Giles Barclay-Beuthin used an AI tool called 'Legal Genius' to draft legal submissions under time pressure. The AI tool, which claimed to be trained exclusively on South African legal sources, generated fictitious case citations that were included in the court documents. Acting Judge DJ Smit discovered that two cases cited in Northbound's heads of argument did not exist while preparing his judgment. When questioned, counsel initially claimed the wrong version was submitted but further investigation revealed additional incorrect citations. Barclay-Beuthin eventually admitted the citations likely originated from the AI tool, explaining he used it due to urgent deadlines and staff unavailability. Senior counsel Arnold Subel apologized and explained he had only conducted a general review without verifying citations. Despite accepting the team's apologies and finding no intent to deceive, Judge Smit referred the conduct to the Legal Practice Council for investigation, emphasizing that professional obligations remain unchanged regardless of tools used. This represents the third known instance in South Africa where AI-generated false citations appeared in court filings, following similar cases involving ChatGPT and the Mavundla case.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed