Former Anthropic researcher Jacob Coxon has warned that some people working on advanced artificial intelligence are “genuinely frightened” by how quickly the technology is progressing. The 27-year-old, who previously worked at OpenAI, said he believes continued development at the current pace creates a serious risk to humanity.
Coxon spoke after his resignation post about AI safety attracted widespread attention. He told the BBC that employees inside AI companies are thinking about the potential consequences of their work in their daily lives, with some considering preparations for instability caused by increasingly capable systems.
Coxon Says AI Labs Feel Trapped in a Race
Coxon said researchers inside AI companies support regulation partly because competition makes it difficult for individual companies to slow development on their own. He welcomed Anthropic CEO Dario Amodei’s recent proposal to slow frontier AI development, but said any effective agreement would also need coordination with China to prevent an international race.
In his essay “We Must Pace the Frontier”, Amodei argued that AI capabilities should advance more slowly so safety research and safeguards have time to catch up. He proposed independent monitoring, coordination among major AI developers and eventual international agreements.
Amodei said one concern is the possibility of highly capable groups of AI agents operating together on the internet. Coxon told the BBC that a scenario involving agents powerful enough to cause large-scale online disruption could be realistic within six months to a year.
Anthropic said it has consistently acknowledged both the potential benefits and risks of advanced AI. The company pointed to its Responsible Scaling Policy, which sets out safeguards and risk-management measures as its models become more capable.
Coxon said the concern extends beyond his own assessment. Anthropic alignment researcher Evan Hubinger has publicly estimated a greater than 10% probability that AI could kill all humans within the next decade, while computer scientist and Nobel laureate Geoffrey Hinton told the BBC that such a probability was “not unreasonable.”
Industry Leaders Call for Slower Development
Amodei’s proposal has received support from OpenAI CEO Sam Altman and xAI founder Elon Musk, who have also backed greater regulation and independent oversight of advanced AI development. Amodei has argued that the objective is not to stop AI research, but to give companies and governments more time to address safety problems.
Not everyone in the technology industry accepts the warnings. Hugging Face CEO Clément Delangue questioned Coxon’s expertise on extinction risk, while Grindr CEO George Arison suggested that dramatic warnings from AI companies could help justify large valuations and investment.
Nvidia CEO Jensen Huang also reportedly disputed Coxon’s characterization of how people inside AI companies view the risks. Faculty CEO Marc Warner, meanwhile, said placing an exact probability on human extinction was extremely difficult but argued that the concerns raised by researchers should be treated as sincere.
U.S. President Donald Trump took a different position on Sunday, saying the United States was leading China in AI and that he wanted to maintain that advantage. Former U.K. Prime Minister Rishi Sunak, now a paid Anthropic adviser, has also expressed concern about losing control of increasingly capable AI systems while remaining optimistic about the technology’s potential benefits.
Featured image credits: Magnific.com
For more stories like it, click the +Follow button at the top of this page to follow us.