Skynet scenario: Anthropic warns of AI that develops itself

Contents

AI company Anthropic has advocated for a global slowdown in the research and development of artificial intelligence. “We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology,” explained Anthropic co-founder Jack Clark and Anthropic researcher Marina Favaro in a blog post.

However, according to the author team, this would require a global coordination mechanism that could ensure a credible slowdown among states and companies. Otherwise, less cautious actors could simply use such a moratorium to catch up technologically or to secure an advantage.

Warning about Skynet

Anthropic bases its proposal on the increasing progress in AI models and outlines the scenario that humans have less and less involvement in their development. AI may eventually be able to develop new AI models independently. Anthropic calls this “full recursive self-improvement.” Such an AI that develops itself could represent significant technological progress but also carries the risk that humanity could lose control.

Using figures from its own company, Anthropic aims to support indications of such a trend. As of May 2026, 80 percent of the code Anthropic integrates into its codebase reportedly came from the Claude model. In February 2025, this share was still in the low single digits. Thanks to AI, developers at Anthropic now deliver eight times more code per quarter on average than in the period from 2021 to 2025.

A Bottleneck Called Human

However, AI is not only accelerating qualitatively. Anthropic claims: The frequency with which employees have to correct tasks from Claude or intervene themselves has steadily decreased – even for open-ended tasks without clear specifications. Many Anthropic employees believe that the code written by Claude was qualitatively worse than human-written code at the end of 2025 but is now on par. “We expect it to be better within the year,” write Clark and Favaro.

And they add: “Once human- and AI-authored code quality reach parity, humans will stop writing code entirely and shift to only reviewing it. But if they can’t review code as quickly as Claude can generate it, human review will become the bottleneck to AI development.” However, they admit that AI, without human judgment and decision-making power, is currently more of a capable assistant. It remains unclear whether today's training methods and architectures can reach this human potential.

Why Now, Just Before the IPO?

The timing of Anthropic's call for a moratorium is puzzling. The company is considered one of the global leaders and only announced on Monday that it had confidentially filed for an IPO with the US financial regulator SEC. This puts the company ahead of its arch-rival OpenAI, which is also pushing for an IPO.

On the other hand, such initiatives also fit well with the image the company cultivates of being a particularly responsible provider. Likewise, emphasizing the danger of its technology is an obviously attention-grabbing part of its product communication. When introducing Claude Mythos, for example, Anthropic initially restricted access to the model to a small circle of US companies and authorities. Mythos is simply too good at detecting security vulnerabilities and therefore cannot yet be made generally available, the company explained. Meanwhile, institutions from Europe are also allowed to access it, and Anthropic plans to present a report on the vulnerabilities discovered with Mythos at the end of June.

In general, the AI industry likes to use superlatives in its communication, which are sometimes retracted in retrospect. For example, OpenAI CEO Sam Altman recently revised his prediction that AI would broadly replace humans and lead to mass layoffs. Altman said he had overestimated the impact on the labor market.

Open Letter Without Effect

It is rather unlikely that Anthropic will achieve a slowdown in such a competitive and highly dynamic market as AI models. It would also not be the first demand of this kind to fizzle out. In March 2023, for example, an open letter calling for a mandatory pause in the development of the most powerful AI models gained traction and warned of serious consequences of the technology for humanity. Over 1000 people from research and business had signed it, including tech luminaries like Elon Musk and Steve Wozniak. The letter clearly had no major impact on AI development in the following years.

In any case, Anthropic intends to organize discussion rounds with representatives from politics, research, civil society, and AI companies in the coming months to discuss the issues raised in the blog post. The company also plans to advance research into possible security systems for a global AI brake. “If such systems existed, we expect that we would slow down or temporarily pause,” the blog post states – but only if other developers at the technological forefront do so in a verifiable manner.

(axk)