Anthropic co-founder warns AI could develop without human input and urges stronger oversight

Anthropic co-founder Jack Clark has warned that artificial intelligence could advance to a point where it starts developing without human input, arguing that society needs to ensure humans remain in control of how the technology evolves, according to the BBC. His comments came as growing parts of the AI industry are debating whether progress should be slowed or paused to reduce risks.

Clark told BBC Newsnight that the concern is not just that AI systems are becoming more capable, but that they may eventually take on more of the work of improving themselves. That possibility has intensified fears among researchers and executives about losing visibility over how advanced systems make decisions or change over time, especially as more companies move toward increasingly autonomous tools.

The warning lands in the middle of a broader shift in the AI business. As TechCrunch reported, the industry’s mood has moved from a “go fast” mentality toward a stronger focus on guardrails and control, driven in part by the rising cost and complexity of building powerful models. The publication described a scramble to manage AI’s “runaway costs,” reflecting pressure on companies to justify the enormous spending behind the current boom.

That tension is also showing up in cybersecurity and enterprise adoption. Bloomberg reported that Rubrik CEO Bipul Sinha said AI agents could bring “100x more risk,” arguing that businesses need to move from trying to prevent every attack to being able to recover quickly when systems are hit. Other Bloomberg coverage this week said AI is already reshaping financial advice, wealth management and broader markets, underscoring how deeply the technology is spreading into the economy.

Anthropic’s own message has also drawn attention beyond the company. According to Business Insider, leading Anthropic figures have suggested it would be good for the world to have the option to slow or temporarily pause AI development, a stance that has prompted debate across the sector. The company’s investors, meanwhile, appear unwilling to treat the AI race as a zero-sum contest; Wired reported that many are backing both OpenAI and Anthropic rather than choosing one side.

The comments come at a moment when AI is both a source of optimism and anxiety. Goldman Sachs’ Christina Minnis told Bloomberg that the investment boom in AI is a “fundamental, generational” force shaping markets, while Fast Company described enterprise AI as still being in an early stage of adoption, with companies figuring out the infrastructure, governance and security needed to make it reliable at scale.

For regulators, companies and researchers, Clark’s warning adds urgency to a question that is no longer theoretical: how to keep human oversight in place as AI systems become more capable, more autonomous and more embedded in everyday business decisions.