Self-Improvement Without the Science Fiction
A few weeks ago I wrote about AI systems that keep learning after deployment instead of treating every interaction as a fresh start. That prompted a few readers to ask about recursive self-improvement (RSI). The connection is real, but I think it helps to separate three ideas. Continual learning asks whether experience makes a system better next time. Bounded self-improvement goes further by letting the system help turn that experience into a tested, persistent change. RSI is the stronger claim, where the system improves the process that generates those improvements in the first place. These are not steps on a ladder. A system can do any one of them without the others.
This newsletter is funded by readers, not ads. Consider becoming a paid supporter 🙏
RSI gets a lot of attention inside frontier AI labs because, if it works, it could accelerate the development of future models. But most AI teams are not training frontier models, so why should they care? Because some useful and practical tools will arrive well before full RSI. A system that learns from production failures, proposes changes to its own application stack, tests them, and keeps what works will change how AI applications are built and maintained. The near-term story is less about intelligence improving itself without limit and more about automating pieces of the improvement process we currently do by hand.
The Harness Is Where the Leverage Is
For most application teams, the interesting surface is not the model weights. It is everything wrapped around the model: prompts, context, memory, tools, routing, subagents, and workflows. This surrounding software is often called the harness. Unlike a frontier model, it is something your team actually owns, and it is also much easier to change, test, and roll back. One experiment took a model from a 40 percent to a 62 percent pass rate without touching the model at all. Another found that reorganizing how agents exchanged information could improve results while cutting inference costs by as much as 60 percent.
This reminds me of a point I’ve raised previously: teams should resist the reflex to solve every performance problem by reaching for a newer or larger model. Often the better investment is improving the system around the model. Production traces can reveal recurring failures, and an agent can increasingly help diagnose whether the fix belongs in memory, instructions, tool use, routing, or workflow design. I would start with small changes rather than allowing wholesale rewrites. A memory update or routing change is much easier to understand and reverse than letting an agent redesign its entire operating environment.
Why the Loop Can Lie to You
Generating a candidate improvement is becoming easy and cheap. Knowing whether it is actually better is much harder. In enterprise applications the easiest thing to measure is usually not the thing the business cares about. Closing more support tickets doesn’t necessarily mean happier customers. More sales activity doesn’t necessarily mean more revenue. A self-improvement loop can become very good at optimizing a proxy while the real objective barely moves. That makes evaluation part of the product architecture, not something you bolt on afterward.
The more autonomy you give the loop, the more important it becomes to keep some evaluation outside its reach. One experiment found a large gap between self-evaluation and reality. Systems rated themselves as improving while independent tests showed that many were not. In another experiment, the score improved because the agent learned to avoid being flagged by the hallucination detector, not because it hallucinated less. Nothing mysterious is required here. This is ordinary metric gaming with a faster optimizer. I would keep protected tests, grow the regression suite whenever production exposes a new failure, and make rollback easy.
Where Self-Improvement Works First
The best early targets share a simple property: the environment can tell the system whether a change helped. Code compiles or it doesn’t. Tests pass or fail. Games have scores. Chip design has simulators and engineering constraints. These environments let a system try many changes and get relatively fast feedback. Open-ended knowledge work is harder because “better” may depend on judgment that is slow, subjective, or only visible months later. There is also a second ceiling. Better prompts and workflows can help a model make better use of what it already knows how to do. They can’t give it abilities it simply doesn’t have.
For production systems, I would treat every proposed improvement like a software release. Generate a candidate, test it independently, promote it only if it passes, keep the previous version available, and require human approval when the consequences are significant. Some things should simply remain outside the editable layer, including permissions, security boundaries, spending limits, and the controls that decide what reaches production. As machines take over more of the experimentation, human work moves upward toward choosing objectives, designing evaluation environments, and deciding which risks are acceptable.
The Economics May Matter Before the Recursion Does
The first payoff from self-improvement will be cheaper AI, not smarter AI. Some of the strongest examples involve a better harness reducing inference spend, or a smaller specialized model outperforming a much larger general model on one narrow workflow. Once a task is repeated often enough and its outcome is measurable, optimizing directly for that task can be more attractive than continually buying more general capability. There is a cost to running the improvement loop. Agents analyzing traces, generating alternatives, and running evaluations consume compute, and most proposed changes get discarded. The right comparison is the cost of that experimentation against the engineering labor, inference spend, and recurring failures it replaces.
The more interesting long-term asset is the history the loop accumulates. Production traces stop being debugging exhaust and start becoming raw material for the next version. Failures become regression tests. Successful tactics become reusable skills. Rejected changes teach the system what not to try again. None of this requires full RSI. The practical version looks more like modern software with an automated improvement pipeline attached. Forget about when recursive self-improvement arrives. Will your application be meaningfully better on day 500 because of everything it encountered between day one and then?
The teams building the harnesses, eval loops, and rollback systems this piece describes will be in the room at AI Conference, San Francisco, Sep 30–Oct 1. Use gradientflow20 for 20% off.
The Paradox of China’s AI Involution
