Anthropic researcher quits over fears of self-improving AI

Jacob Coxon, a researcher who had worked at both Anthropic and OpenAI, resigned from Anthropic and said the companies are pursuing more powerful systems faster than they can make them safe. His concern is not today’s chatbots, but the prospect of AI that can help design its own successor, potentially accelerating its capabilities beyond human oversight. That kind of recursive self-improvement does not yet exist, and the timing and scale of its risks are deeply uncertain. But Evan Hubinger, an Anthropic researcher focused on keeping AI aligned with human intent, publicly backed Coxon’s broader concern and put his own estimate of an AI-driven human-extinction risk over the next decade above 10%. The episode exposes a central tension in the AI boom: people inside the leading labs fear competitive pressure could outrun the safeguards.