// ENGADGET — INTELLIGENZA ARTIFICIALE
What happens when AI starts self-replicating?
Anthropic warns that future is soon, but the threat is less novel than it seems.
Anthropic's Frontier Red Team recently published research suggesting that small AI models may be only months away from replicating and spreading across insecure devices, creating slow but hard-to-kill autonomous botnets. AI companies are no strangers to playing up the world-ending potential of their products as a form of marketing, but in this instance, experts Engadget spoke to broadly agree small AI models could become capable of self-replicating in the near future. However, the actual threat they pose may be less novel than it appears.
"It's maybe a little earlier than I would have said. I would have predicted something more like summer of next year as the time I'd be really worried about this," says Chris Canal, the CEO of EquiStamp, a company that works with AI labs, safety organizations and governments to verify and evaluate models. "Anthropic very likely has employees with information about the frontier of research that I don't have access to. So they very well could be making better predictions." Canal attributes the recent pace of model advancement to the fact that employees at Anthropic and other frontier labs are using their own AI tools to do machine learning research.
"I think [self-replicating models] are in the future. I think they're real. I think we'll see them," says Michael Bailey, a professor at the School of Cybersecurity and Privacy at Georgia Tech. "Whether they'll take on the exact form and shape, and have the same consequences, that [Anthropic] is pitching is a completely open question."
Bailey has been studying internet worms since the early 2000s, and more recently, botnets. In 2017, he co-wrote Understanding the Mirai Botnet, which Anthropic cites as part of its own research. Mirai allowed a group of cybercriminals to create a large botnet of IoT devices they later used to carry out a DDoS attack against Sony's PlayStation Network. In the process, they ended up affecting Dyn, one of the largest DNS providers in the US. In turn, the attack broke much of the internet in 2016, knocking Amazon, GitHub, PayPal, Reddit and Twitter offline for many users across the US and parts of Europe. The financial fallout of the attack was significant, with Sony estimating it lost about $2.7 million in net revenue.
"What Mirai showed is that you could compose these botnets — and have them be useful — out of everyday devices: cable routers, IP-enabled cameras, these kinds of things," says Bailey. "We thought those devices weren't useful to an adversary, either as targets or as sources of computation for launching attacks. In fact, they're actually quite effective at doing those kinds of things."
A decade later, Anthropic argues AI worms could represent a meaningful evolution of Mirai and other hard-coded botnets like Kimwolf because they could figure out unfamiliar devices on their own. Bailey says adaptability itself isn't new. Mirai already worked across "heterogeneous" devices, from routers to IP cameras running on different chip architectures. But that adaptability was finite: each new architecture required human work, at minimum recompiling the malware. What's novel, Bailey says, is the prospect of AI automating that work, though he doesn't think Anthropic demonstrated it "in a compelling way."
As a result, Bailey suggests more capable AI models will "redistribute" who does the work of creating botnets. "It's not going to do new stuff," he says. "It's going to change the economics between attackers and defenders." While Anthropic showed that a model could hop between devices and set up inference on each, it did so in an emulated network where the "exploit" involved was trivial. Additionally, Bailey takes issue with Anthropic's self-replicating claim.
"The framing is somewhat of a misnomer, because they're doing things like sharding. Not that that's bad, but it's not like a worm or a virus. It's not full propagation," he says.