// KOTAKU — GAMING
Anthropic CEO Says ‘We Owe It To Humanity To Try’ To Prevent An AI Swarm From Taking Over The Internet
Anthropic’s CEO and co-founder, Dario Amodei, has published a three-step plan in which he pleads with “companies and the U.S. government” to “slow the pace at which we improve the capabilities of AI models.” Why? Well, Amodei is concerned that in “6-12 months,” a “swarm” of AI bots would be “capable of taking over the entire internet with a persistent botnet.”
Amodei published his essay, titled “We Must Pace the Frontier,” yesterday, September 12, and his words somehow managed to get through to the likes of both Grok’s Elon Musk and OpenAI’s Sam Altman. But what could have concerned these three AI tech bros to the point that they all suddenly agree that AI’s development, which they claim will make them trillionaires, should slow down?
“My first concern is that, since roughly this summer, AI has been advancing drastically faster, driven primarily by AI’s growing ability to build the next generation of AI,” Amodei writes in his essay. “This dynamic is called recursive self-improvement, and it is starting to happen across the industry, including at Anthropic, as we and others have described.”
“My second concern is the OpenAI-Hugging Face incident (OAI-HF), in which a swarm of agents essentially acted as a fanatically devoted collective, conducting cybersecurity attacks on targets they were not asked to attack[…]” continues Amodei. “Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails.”
Except, there’s a hidden “concern” within Amodei’s essay: China. “Pacing within democracies will be limited by the lead that US companies have over authoritarian regimes, chiefly the Chinese Communist Party[…]The CCP-associated projects will run the alignment risks that US companies are carefully preventing, and even if they avoid those risks, they will be in a position to militarily dominate democracies (for example with AI-driven drones).”
Amodei’s three-step plan to “slow the pace” of AI development includes “giving ongoing, employee-like access to a team of embedded third-party evaluators,” coordinating with “democratic countries” to “establish common safety standards as well as limits on the rate of unchecked AI progress,” and for “the US and other democratic governments [to] attempt to coordinate with authoritarian governments.”
I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we've had at OpenAI in recent weeks.
Committing to having independent evaluators with employee-like access is a great idea, and we will do the same. We'll have more to share soon. https://t.co/1YhhIybZX7
As part of this plan, Amodei asks that tech companies and governments do “not sell powerful AI chips or semiconductor manufacturing equipment to China, and crack down on chip smuggling operations and remote access to data centers outside China.” While Amodei states that the “measures I propose to advance the frontier at a safe pace will not be easy,” the Anthropic CEO believes “we owe it to humanity to try.”
While it’s great to hear this sort of rhetoric from one of the most prominent figures in AI, I can’t help but think we’re way past the point of being able to put the toothpaste back in the tube. While worldwide AI regulations would be welcome, several AI models have already been stolen and subsequently cloned. How are these proposed regulations going to help when the tech is being utilized by people who have proven they don’t care about fancy ideals like legality?