What Happens When AI Learns To Improve Itself?

The most dramatic leap in AI could come when humans are no longer doing all of the improving.

That is the idea behind recursive self-improvement or RSI: machines that can help design, build and improve increasingly advanced versions of themselves. It could become one of the most important developments in the history of artificial intelligence.

It could also be one of the most dangerous.

In theory, recursive self-improvement could dramatically accelerate AI progress, helping us tackle scientific, healthcare and environmental problems that are currently beyond our reach. But some researchers fear it could eventually trigger an “intelligence explosion”, where machines become smarter so quickly that humans struggle to understand, predict or control them.

Others are far more skeptical, arguing that RSI remains largely theoretical and is sometimes used alongside terms such as AGI and agentic AI to generate hype.

So how close are we to genuinely self-improving AI, and should we be excited or worried? To answer that, we first need to understand what recursive self-improvement actually means.

So What Is It?

Solving a problem with AI involves identifying the problem, building a tool, processing data and then evaluating the results to see what needs to change to find the solution.

In a standard AI workflow today, humans own the first two steps, while processing the data and tweaking the results is handled by machines.

A system capable of RSI automates the entire process end to end. When this idea was first put forward during the earliest days of AI research in the sixties, it was predicted that it would lead to an “intelligence explosion”, with machines becoming exponentially smarter and more capable.

This sounds great if your job is simply to create more and more powerful AI, but a lot of people think it might not be a great idea.

To understand why, it helps to visualize intelligence as working on different scales. Just as it’s impossible for an ant to understand how our brains work, we might be simply unable to comprehend the plans and decisions of a superintelligent AI.

This would leave us with no way of knowing whether we could trust it, or whether its plans are aligned with our own.

And alignment is already a concern, with Anthropic’s Evan Hubinger telling Time magazine this year that its ability to determine what AI’s true intentions are is degrading as models become more powerful.

Of course, for the companies building frontier models, there are huge incentives to get there first. RSI could potentially accelerate progress on projects using AI to tackle major scientific, healthcare and environmental problems, as well as improve the intelligence and usefulness of the AI tools and platforms we use every day.

Are We There Yet?

Not quite, it seems.

RSI is undoubtedly a major research priority at frontier AI labs like OpenAI, Anthropic and Google, and pilots and small-scale experiments involving end-to-end RSI undoubtedly exist.

But there’s no claim or evidence that the world’s most powerful AIs are improving themselves without human intervention yet.

They have, however, started measuring their progress towards it. Anthropic disclosed this year that up to 80 percent of the code added to its codebase is now written by Claude. Importantly, however, while this makes it clear that Claude can write its own code, there’s no indication yet that it can make high-level decisions.

Decisions around what the code should actually do, how it should work, whether it’s good enough to go into production, and how to make sure it doesn’t cause harm are all overseen and ultimately taken by humans.

Neither OpenAI nor Google have claimed that their own frontier models are at the stage where they can build and improve themselves without human oversight.

In fact, Google’s AI head, Demis Hassabis, describes what’s happening now as “soft self-improvement,” with AI agents prioritizing helping humans work smarter rather than developing their own superintelligence.

Others aren’t even willing to go that far, describing what’s been done so far as simply enabling AI to code faster, rather than become genuinely smarter.

What Does This All Mean?

Getting to the point where machines genuinely are churning out increasingly smarter and more powerful machines on a production line may require a change of thinking, not by us, but by the machines.

Recent research found that while AIs tasked with working fully autonomously often greatly exceed human ability at solving technical problems, they fall short at judgment and creativity. Both qualities are essential for machines to truly work without human input.

In his own analysis, Anthropic co-founder Jack Clark has written that “There’s a certain absence of valuable, intuitive creativity in today’s AI systems …that might prevent [them] from being good researchers.”

This means that for now at least, humans are going to stay in the loop.

So, while RSI is a genuine goal, with consequences that we might not be able to fully comprehend, it’s still some way off. Simply because there’s still a big difference between the way they think and the way we think, and that’s likely to be true for some time yet.