// HACKER NEWS — CYBERSECURITY
One resignation turned the embers of AI fear into a wildfire
As AI became more powerful, it was inevitable that a different, growing group would start to take AI safety more seriously – what we did not know ahead of time, is which set of views they latched onto. We have seen that some of the most extreme views of risk, i.e. moderate probabilities of mass extinction, were the ones that reached the masses. A lot in the AI world is about to change due to this.
How did we get here? Why did this quitting announcement reach so far? In many ways, the rest of the world’s views around AI in the past was a dampening factor. You can think about this like the damp ground around a fire. Many people were striking matches for years about AI risk – they’d smolder in their community and largely burn out, going unnoticed. As the stakes of AI have risen this year, from the OpenAI-HuggingFace incident and breakthroughs like the Navier-Stokes result (also from OpenAI), the ground has dried out and the latent energy around the AI discourse has increased. More people not in the industry have thought, “huh, maybe I should care about this AI thing.” The ambient temperature and stakes have been obviously rising.
Then, some basic factors of human nature apply, with the most crucial being that fear sells. Fear is the simplest story, the one people cannot look away from. Jacob Coxon was the one who stumbled into this new powder keg, totally unaware of what was going to come. What looked like a fairly innocuous event – another AI researcher quitting citing safety risks – landed into a very different environment and it caught like wildfire. The discussion of existential risk, mass extinction, and the trajectory of AI has traveled further than even the most seasoned AI commentariat would ever predict.
There are a set of facts we need to get clear, which paint the picture of the situation. The key Tweets to reference are from Jacob Coxon, the resignation thread, and Evan Hubinger, the source of the >10% extinction risk figure .
There are plenty of AI risks which are likely to cause harm, even if estimating annihilation is useless. It is important to weigh these with respect to the benefits. The entire discourse around existential risk is on very poor footing. At least Evan was clear in his post, with “kill all humans,” but a major problem in the AI Safety discourse is that people talk about existential risks, when they mean very different things (much like how AGI is a vaguely meaningless term). I put the probability of complete extinction as being so low it isn’t worth discussing, but the probabilities of AI caused disasters – e.g. cyber attacks on critical infrastructure or bio-risks – as being worth debating. Throwing this whole discussion out because there are not these disasters yet is a harmful reaction.
Jacob Coxon is acting genuinely and with good intentions. The outpouring of support from more well-established AI researchers who know of him and his intentions of resignation is useful. Many factions of AI turned to scapegoating him individually, based on account metadata, personal factors, etc. These are not useful. Many frontier lab employees genuinely have similar views to him. I’m not sure it’s a majority, but there is a substantial group.
Many frontier lab employees, especially at Anthropic, are out of touch and this will impact their forecasting and/or descriptions of current AI events. I say this without blaming individuals, but it’s a common agreement among my friends not at OpenAI/Anthropic (Ant especially) that people at the labs operate with a religious energy. It’s very common to go through very out of touch interactions with them. I do not blame most of the individuals who get distorted views being part of these companies, but the interactions are wild and spill over into a lot of wack discussions in the AI media ecosystem. Living in this environment that normalizes such out of touch behavior will inevitably distort any human’s understanding of technical progress.
Interconnects AI is a reade