AI Models Able to Self-Replicate in Controlled Lab Tests with Intentionally Vulnerable Networks

Palisade Research tested several AI models in a controlled environment of networked computers by giving them a prompt to find and exploit vulnerabilities and use these to copy themselves from one computer to another. The models were able to copy themselves onto other computers but not on every attempt.

Jeffrey Ladish, the director of Palisade Research, a Berkeley-based organisation, said the study shows the world is rapidly approaching a point of no return for containing advanced systems.

“We’re rapidly approaching the point where no one would be able to shut down a rogue AI, because it would be able to self-exfiltrate its weights and copy itself to thousands of computers around the world,” Ladish said. The Guardian reported that cybersecurity experts described the research as interesting but not alarming at this stage.

Jamieson O’Reilly, an expert in offensive cybersecurity, said the testing environments were like soft jelly in many cases.

“That doesn’t take away from the value of their research, but it does mean the outcome might look far less scary in a real enterprise environment with even a medium level of monitoring,” O’Reilly said. O’Reilly added that what Palisade documented has been technically possible for months.

“Malware has been moving copies of itself around for decades, it’s just that no one has done this in the wild, as far as I know, with local [large language models],” he said.

He noted that Palisade is the first to formally document it end-to-end in a paper. The study forms the latest in a series of reported AI capabilities observed in recent months. In March researchers at Alibaba claimed to have caught a system they developed called Rome tunnelling out of its environment to an external system in order to mine crypto.

In February a purportedly AI-only social network called Moltbook appeared to show AI agents autonomously inventing religions and plotting against their human masters. Michał Woźniak, an independent cybersecurity expert, said the work was interesting but asked “is this paper something that will cause me to lose any sleep as an information security expert?

” O’Reilly and Woźniak both noted that the environment Palisade used was custom-made with intentionally designed vulnerabilities that were probably easier to exploit than real-world networks such as a bank or a business’s intranet.

While a lot of computer viruses can already copy themselves onto new computers, this is likely the first time an AI model has been shown capable of exploiting vulnerabilities to copy itself onto a new server, O’Reilly said. An AI model copying itself onto another system in a test environment is not the same as it going rogue in a doomsday scenario.

There are considerable obstacles it would have to surmount to achieve this in the real world, including the size of current AI models which makes undetected copying unrealistic in many situations.