Experts Warn of ‘Hallucinatory Hijack’ Scheme Attacking PC Developers Using AI

Cybersecurity researchers have warned of a new method by which hackers could abuse the “hallucinations” that artificial intelligence uses to develop software. The theoretical scheme is called “slopsquatting.”

Image source: socket.dev

Generative AI systems, including OpenAI ChatGPT and Microsoft Copilot, are prone to hallucinations — when the AI ​​simply makes up something that doesn’t correspond to reality and passes it off as factual information. It might attribute words to a person that they didn’t say; invent an event that didn’t happen; or, when developing software, refer to an open-source product that doesn’t exist.

With the spread of generative AI, many software developers have come to rely heavily on it when writing code. Chatbots write the code themselves or offer third-party libraries to the developer to include in the project. It is noteworthy that when hallucinations occur, the AI ​​can repeatedly invent the same non-existent package. If you find a request that provokes a response with a hallucination, experts from the Socket company have established, and repeat this request ten times, then in 43% of cases the AI ​​will again refer to a non-existent software product, and in 39% of cases it will not remember it. In general, 58% of the packages invented by the AI ​​appeared more than once per ten requests, and attackers can try to play on this.

The scheme exists only in theory, but there is nothing stopping hackers from identifying such a pattern in failures and creating a non-existent software package that the AI ​​persistently refers to — and injecting malicious code into this package. Having received a recommendation from the AI, the developer will open one of the most popular platforms, for example, GitHub, find the specified product and deploy it in the project, not knowing that it is malware. At the moment, there are no registered incidents using the “hallucinatory hijacking” scheme, but it is probably only a matter of time before they occur. The best way to protect yourself is the same as always — caution.