New York City's AI-powered business chatbot provided false and illegal advice to users, including telling landlords they could discriminate against tenants with housing vouchers and that employers could take workers' tips.
In October 2023, New York City launched an AI-powered chatbot designed to provide business owners with information on starting and operating a business in the city. The chatbot, powered by Microsoft's Azure AI services and trained on over 2,000 NYC Business web pages, was intended to help with compliance, regulations, and best practices. However, five months after launch, testing by The Markup revealed the chatbot was providing dangerously inaccurate legal advice. The bot incorrectly told users that landlords could discriminate against tenants with Section 8 housing vouchers, that businesses could operate cash-free, that employers could take workers' tips, and that there were no restrictions on rent amounts or tenant lockouts. These responses directly contradicted New York City laws, including the illegal discrimination based on source of income since 2008, mandatory cash acceptance since 2020, and various worker protection laws. The chatbot provided inconsistent responses, sometimes giving correct answers but more frequently providing illegal advice. Business owners and advocacy organizations reported discovering these inaccuracies, with the NYC Hospitality Alliance noting the bot could create 'massive liability' for businesses following its incorrect legal guidance.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed