Researchers have discovered a concerning vulnerability in ChatGPT that could potentially be exploited by attackers to propagate harmful code packages. This particular weakness stems from ChatGPT's tendency to provide inaccurate information, which could be leveraged to introduce malicious software and Trojans into trusted applications and code repositories such as npm, PyPI, GitHub, and various others. 

This represents a substantial threat to the software supply chain. In a recent blog post, researchers from Vulcan Cyber's Voyager18 research team have shed light on a concerning method employed by threat actors, known as "AI package hallucinations." This technique exploits ChatGPT's capability to generate recommendations, leading to the creation of seemingly legitimate code packages that contain malicious elements. 

Developers who interact with the chatbot may unknowingly download these packages and integrate them into their software, which can subsequently be widely distributed. This discovery highlights the potential risks associated with the misuse of ChatGPT and its impact on software security. 

In the realm of artificial intelligence, the term "hallucination" refers to a response generated by AI that appears reasonable but falls short in terms of accuracy, bias, or outright falsehood. This phenomenon arises due to the nature of ChatGPT and similar large language models (LLMs) that form the foundation of generative AI platforms. 

When posed with questions, these models rely on information sourced from the vast expanse of the Internet, which includes various types of data such as sources, links, blogs, and statistics. However, the training data available to these models may not always be reliable or of the highest quality. Consequently, the AI's responses can be influenced by this imperfect training data, leading to hallucinations that do not align with factual information. 

In the blog post authored by Bar Lanyado, the lead researcher at Voyager18, he highlighted that LLMs such as ChatGPT possess extensive training and exposure to vast amounts of textual data. As a consequence, these models have the ability to generate responses that may appear plausible but are actually fictional. 

Furthermore, he said that LLMs have a tendency to extrapolate beyond their training, potentially leading to the production of responses that seem credible but lack accuracy. 

In their demonstration, the researchers conducted an experiment utilizing ChatGPT 3.5 to validate their concept. They constructed a scenario where an attacker posed a coding problem to the platform, requesting a solution. As a response, ChatGPT generated a set of packages, including some that were non-existent, indicating they were not available within a reputable package repository. 

This practical demonstration served to illustrate how the platform could generate misleading and potentially malicious package recommendations. According to the researchers, the fabricated code packages produced by ChatGPT could serve as a novel avenue for attackers to distribute malicious software, bypassing conventional techniques like typosquatting or masquerading. 

By presenting these fabricated packages as genuine recommendations from ChatGPT, attackers can exploit the trust developers place in the platform's suggestions. Consequently, there is a significant risk of malicious code infiltrating legitimate applications and code repositories, thereby posing a major threat to the software supply chain. 

According to the researchers, detecting malicious packages can be challenging, especially when threat actors employ obfuscation techniques or create functional Trojan packages. However, developers can take preventive measures by thoroughly validating the libraries they download. It is crucial to ensure that these libraries not only perform their intended functions but also aren't cleverly disguised Trojans posing as legitimate packages, as highlighted by Lanyado. 

Since its release in November, ChatGPT has gained popularity not only among users but also among threat actors who exploit it for cyberattacks. In the first half of 2023, security incidents have included scams targeting user credentials, theft of Chrome cookies through malicious ChatGPT extensions, and phishing campaigns utilizing ChatGPT as bait for malicious websites. 

While some experts argue the security risk may be exaggerated, the researchers emphasized that the rapid adoption of generative AI platforms like ChatGPT has indeed introduced potential security concerns due to their integration into daily professional activities and workload management.