Hallucinated Software Packages with Potential Malware Downloaded Thousands of Times by Developers

Multiple generative AI models including GPT-3.5-Turbo, GPT-4, Gemini Pro, and Command were found to hallucinate non-existent software package names when providing coding assistance. Security researcher Bar Lanyado from Lasso Security demonstrated this vulnerability by creating a fake Python package called 'huggingface-cli' after observing AI models repeatedly recommend it. The package was uploaded to PyPI in December and by February, major companies including Alibaba had incorporated installation instructions for this fake package into their GraphTranslator project documentation. Research studies found that commercial AI models hallucinate package names about 5.2% of the time while open source models do so 21.7% of the time. In one comprehensive study of 16 LLMs generating 576,000 code samples, researchers identified 205,474 unique hallucinated package names. The fake 'huggingface-cli' package created by Lanyado was downloaded thousands of times by developers following AI recommendations. While Lanyado's package was benign, the incident demonstrates how malicious actors could exploit this vulnerability by creating malware-laden packages using AI-hallucinated names, a technique dubbed 'slopsquatting' by security experts.