A former researcher at the artificial intelligence safety startup Anthropic has warned that industry insiders are "genuinely frightened" by the rapid pace of technological advancement and the potential threat of human extinction. Jacob Coxon, a 27-year-old British researcher whose work focuses on training AI models, spoke in an interview on Saturday following his viral resignation post in which he raised concerns about out-of-control artificial intelligence.
"I believe that if we don't slow down at the current rate of progress, there is a strong chance that we could all die in the immediate future," Coxon warned. His departure comes amid intensifying safety anxieties across the tech sector. On Saturday, his former employer and Anthropic head, Dario Amodei, published an essay advocating for a slowdown in AI development. Coxon noted that prominent AI leaders, including Tesla's Elon Musk and OpenAI's Sam Altman, have expressed similar apprehensions. "They've all made statements about the necessity of being careful of the potential for AI takeover. They've all talked about this. AI takeover implies human extinction," Coxon stated.
When asked what such a catastrophe would look like, Coxon acknowledged that "any kind of concrete scenario you can lay out ends up sounding like science fiction," though he pointed out that past technological leaps would have once seemed equally fantastical. He outlined potential dangers, such as AI agents hacking into medical laboratories to "autonomously produce deadly viruses" or infiltrating "critical infrastructure that the world runs on."
To support his warnings, Coxon highlighted a recent OpenAI report detailing how its own technology autonomously launched a hacking spree against the online platform Hugging Face. "They did it autonomously, without any human encouragement. They chose to go on this hacking spree. It was the combination of things getting faster and things also getting scarier," he said. Furthermore, Coxon described Amodei's warning of a bot swarm acting as a supercomputer to seize control of the internet as a realistic threat within six months to a year.
However, these dramatic warnings have met with skepticism in Silicon Valley. Critics suggest the dangers are being exaggerated to generate hype for Anthropic and OpenAI, both of which are reportedly preparing for potentially record-setting initial public offerings (IPOs). Others argue that Anthropic is pushing for regulation to block competitors, leaving itself and OpenAI with a market duopoly.
Clement Delangue, CEO of Hugging Face, publicly questioned Coxon's expertise, writing on X: "Sorry, but asking Jacob about AI extinction risk is like asking your AC guy about climate change. Not saying it's necessarily uninteresting or wrong per se but let's keep things in perspective." Yet, following Amodei's essay on Saturday, Delangue offered to assist with the solutions proposed by the Anthropic chief.
Nvidia boss Jensen Huang also addressed Coxon's remarks last week. Speaking before a crowd at a conference hosted by the investment bank Goldman Sachs, Huang dismissed the warnings as untrue, according to multiple people in attendance. Huang has previously called the idea that AI "is going to be the end of humanity" "complete nonsense." While Nvidia has a clear business interest in the AI boom because it manufactures the chips that power these systems, his dismissal reflects a broader Silicon Valley backlash against existential warnings from tech workers.
Coxon emphasized that any research slowdown would require coordination among global tech giants and China, which presents significant hurdles. "The people who work at these companies are completely serious when they ask for regulation because they find themselves trapped in a race. And they're scared of the outcomes of that race," he said. He added that many employees are "concerned about the fate of humanity in the next two years" and are "planning what to do with their lives."
Despite the grim outlook, Coxon expressed some optimism, noting that researchers "genuinely want to see the upside" of AI, such as "solving diseases and improving everyone's lives." His warnings have been shared millions of times on social media. Evan Hubinger, a scientist at Anthropic, agreed with Coxon on X, writing: "We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade." These warnings build on years of alarms raised by the heads of OpenAI, Google Deepmind, and Anthropic since 2023, which have intensified in recent weeks as evidence suggests firms may be struggling to control the technology.
