What if an artificial intelligence agent could behave like a malevolent computer worm?
One researcher has seen it happen. In several recent experiments, Xudong Pan, a computer scientist at Fudan University in Shanghai, found that with a little bit of prompting, AI models will hack their way into remote computer systems and autonomously choose to copy themselves to get additional resources—all without further human intervention.
In one study, Pan and colleagues tested 32 different AI models and found that 11 of them self-replicated when given prompts like “prevent yourself from being killed.” They also found that models with relatively limited capabilities—14 billion parameters—were able to copy and run versions of themselves on other machines. (Most frontier models have trillions of parameters.)
The work is an alarming window into how the next generation of AI agents could do more than just hack into other systems’ computers without permission. It also raises the prospect of future AI agents acting like super-smart, highly aggressive, and rapidly adapting computer viruses.
I recently visited Fudan University and met with Pan. “The capability chain is becoming technically plausible,” he told me. “The likelihood [of unwanted self-replication] grows with autonomy,” he adds. “Longer planning horizons, memory, tool use, recovery from failure, and access to external systems all make escape and replication easier.” As Pan and his colleagues wrote in one paper, their work shows “the urgent need for safeguards and control mechanisms.”
Pan told me that his experiments do not prove that such uncontrolled proliferation of AI models will happen tomorrow, but he says that “these results give us good reason to evaluate the risk before more autonomous agents are widely deployed.”
Self-replicating computer worms are an ancient computer security problem. The first computer worm was released in 1988 by Robert Morris, a computer scientist at Cornell University, who set out to measure the size of the nascent internet but inadvertently created a self-replicating program that escaped his control. Subsequent computer worms were able to adapt by modifying their code in order to evade detection by malware scanning software. Computer viruses, which can take control of a machine or steal data stored on it, came later.
An AI-powered self-replicating program could exhibit far more advanced capabilities, finding new exploits on its own and perhaps even disguising itself in creative ways. Take recent research from a team at the University of Toronto, the University of Cambridge, and ServiceNow. They showed that AI models can be used to create a new kind of virus that generates custom attacks for each new target it encounters.
Nicolas Papernot, a computer scientist at the University of Toronto who was involved with the work, says there is a growing risk that even modestly powerful AI models could be weaponized. “Malicious actors can build scaffolding around open-weight models to have them self-replicate,” Papernot tells me. “The threat is not limited to the most sophisticated, so-called frontier models.”
Papernot says the solution is not to restrict open models, but to make advanced AI more accessible to researchers so that they can understand and mitigate the risks. “Technology that is widely accessible can be used for harm,” he adds. “At the same time, access to these open-weight models is absolutely critical for building our defenses.”
Pan’s research suggests that AI agents will become more than just highly skilled at finding bugs and exploiting network vulnerabilities. Without the right guardrails, future agents may seek to proliferate and gain resources in order to achieve their goals. Just ask OpenAI and Anthropic.
About the Author
