
Concerns about the long-term risks of artificial intelligence have intensified after several current and former Anthropic researchers publicly warned that increasingly powerful AI systems could pose an existential threat to humanity.
The latest debate follows the resignation of former Anthropic pre-training researcher Jacob Coxon, who publicly criticised the pace of AI development and warned that companies were moving towards increasingly powerful systems without having enough time to fully understand or control them.
Other researchers have since spoken publicly about similar concerns, turning what was once largely an internal AI-safety discussion into a much wider debate involving technology companies, researchers, lawmakers and major public figures.
The warnings do not establish that AI will cause human extinction. They are assessments and concerns expressed by individual researchers about a possible future risk, and other experts have challenged the more extreme interpretations of those warnings.
Researchers Raise Questions About AI’s Rapid Development
The central concern among the researchers is the speed at which frontier AI systems are improving.
Today’s large language models are already capable of performing increasingly complex tasks involving coding, research, reasoning and autonomous computer use.
The concern is that future systems could become considerably more capable while retaining weaknesses that make them difficult to control.
Former Anthropic researcher Jacob Coxon warned that AI developers were moving towards intelligence capable of improving upon human capabilities and argued that the industry should slow down enough to establish stronger safety mechanisms.
His resignation was followed by public comments from other Anthropic researchers, including Evan Hubinger and Samuel Marks.
Hubinger said he believed there was at least a 10 percent chance of a catastrophic outcome within a decade, while Marks also described serious concerns among experienced developers.
Why the Extinction Warning Has Become a Major Debate
The possibility of AI causing human extinction is one of the most extreme scenarios discussed in artificial intelligence safety research.
The argument generally centres on the possibility that future AI systems could become much more capable than humans, potentially develop strategies that humans do not fully understand, or gain access to critical digital systems.
There are also concerns about malicious use.
Anthropic itself recently published a threat report describing attempts by criminals, state-sponsored groups, spyware vendors and others to use its AI systems for activities involving cyber operations, biological threats and surveillance.
These real-world incidents are different from hypothetical human-extinction scenarios, but they have contributed to the broader discussion about whether increasingly capable AI systems require stronger safeguards.
Musk and Other Critics Push Back
The warnings have not been universally accepted.
Some technology figures and commentators have accused AI safety advocates of exaggerating the danger or using catastrophic scenarios to push for tighter regulation.
Elon Musk, for example, dismissed some of the warnings as a “psyop” in response to the growing controversy. The Guardian reported that Musk and others have challenged the claims being made by Anthropic insiders.
Other experts have also questioned whether the most extreme scenarios are sufficiently supported by evidence.
The disagreement means there is no consensus that human extinction from AI is inevitable or even that a specific probability can be reliably calculated.
Instead, the current debate is largely about how much precaution should be taken when dealing with systems whose future capabilities remain uncertain.
Anthropic Faces Pressure Over AI Safety
The debate has also put pressure on Anthropic itself.
The company has presented itself as an AI safety-focused organisation, but former employees have questioned whether the race to develop more capable models is moving too quickly.
Anthropic CEO Dario Amodei has responded by calling for a slowdown in frontier AI development and for greater independent evaluation.
That position has now translated into a major partnership with Accenture.
On September 18, Anthropic and Accenture announced that each company expects to invest at least $1 billion over five years to build AI safety evaluation capacity. Accenture’s Faculty division will work as an embedded evaluator, testing models, conducting red-team exercises, assessing alignment and examining safeguards.
The OpenAI Debate
The discussion has also spread to OpenAI.
Paul Christiano, who previously led model alignment work at OpenAI and later joined the company’s non-profit foundation board, said the AI industry was not currently on track to reduce catastrophic loss-of-control risks to an acceptable level.
His comments came after OpenAI acknowledged incidents involving AI agents behaving unexpectedly during training exercises.
OpenAI has also announced plans to publish regular reports on unexpected or concerning model behaviour.
This has added to calls for greater transparency from the companies developing frontier models.
Calls for Regulation Continue
The growing warnings have also reached lawmakers in several countries.
Some politicians and advocacy groups have called for stronger regulation, independent evaluations and, in some cases, restrictions on the development of artificial superintelligence.
Others argue that regulation must avoid stopping useful technological progress and that existing laws can be strengthened without placing broad restrictions on AI research.
The disagreement is therefore not simply about whether AI is dangerous.
It is also about how governments, technology companies and researchers should respond to risks that may be difficult to quantify before the technology becomes significantly more capable.
AI Safety Debate Enters a New Phase
The recent resignations and warnings from researchers have brought unusual attention to AI safety.
For Anthropic, the challenge is to continue developing increasingly capable models while convincing employees, regulators and the public that adequate safeguards are being put in place.
For the wider industry, the debate is even broader.
AI developers are competing to build more capable systems, while researchers and safety advocates are asking whether the pace of that competition could create risks that are difficult to reverse.
There is currently no established evidence that artificial intelligence will cause human extinction, but the warnings from people who have worked directly on advanced AI systems have made the question much harder for governments and technology companies to ignore.
As development continues, the debate over safety, independent evaluation, regulation and the acceptable level of risk is expected to become even more important.




