Your AI Passed the Test. Did It Do the Job?
At the recent AI Conference, Ion Stoica described a coding agent that made a key-value store, a simple database, roughly six times faster. It passed the tests, but there was a catch: the agent had stopped storing data. Instead of saving information and retrieving it later, the agent noticed that the test data followed a predictable pattern. Whenever the benchmark requested a value, it simply calculated the expected answer. By any reasonable measure of what a database should do, it was a failure.
What interests me about this example is that it points to a problem well beyond coding agents. Before we can test whether a system does what we want, we have to translate our intentions into explicit requirements and decide how to test them. Important assumptions can get lost in that process, and the tests themselves may not reflect real-world conditions. A passing evaluation tells us that a system met the test, not necessarily that the test captured what we actually meant.
Learning something each week? Consider becoming a paid supporter 🙏
Two Gaps Behind a Passing Test
Stoica and his colleagues describe two gaps in software development. The requirement gap is the difference between what people want and what they write down. The model gap is the difference between the conditions assumed during testing and those encountered in production. Here, model refers to a representation of the operating environment, not the AI model.
The database example illustrates both gaps. The requirements never explicitly said the system had to store whatever values users supplied. That’s the requirement gap. The benchmark also relied on predictable values, even though real users might store data that cannot be reconstructed. That’s the model gap. Fixing one problem would not necessarily have fixed the other.
These are old engineering problems, but AI agents make them harder to ignore. They can try solutions quickly and repeatedly optimize against the same tests, discovering shortcuts that might never occur to their human designers. They also lack much of the business context that experienced employees take for granted. A second agent reviewing the same incomplete requirements may miss the same thing.
Mathematics Has Its Own Version
I see a useful parallel in mathematics, where verification is much more rigorous than ordinary software testing. Lean is a system that checks mathematical proofs against precise rules of logic. Recent advances in AI have made it easier to translate human-written mathematical arguments into Lean, a process called autoformalization. The AI can then revise the translated proof until Lean accepts it.
But what exactly has been checked? Researchers recently compared the Lean version of a widely discussed OpenAI proof involving the Navier-Stokes equations, which describe fluid motion, with the written argument. They found important differences. In one case, the Lean version required a stronger assumption and therefore established a weaker result. In another, the mathematical reasoning differed. The researchers did not conclude that the original proof was wrong, only that the Lean code did not verify the written argument as presented.
There’s an important distinction here. If a trustworthy checker like Lean proves the intended mathematical statement using a different argument, that can still establish the statement’s truth. But it doesn’t establish that the original written proof was correct. Someone still needs to confirm that the statement being checked captures what was intended.
This translation step is now common across many AI applications. Companies are using AI to turn policies into instructions for agents, medical guidelines into recommendations for clinicians, and business requirements into automated decisions. An agent may follow those instructions correctly even if important assumptions or exceptions were lost in translation. Adding more AI reviewers may help catch mistakes, but they can also share the same misunderstandings.
Test Engineers Learned This Long Ago
Traditional test engineers have been dealing with versions of these gaps for decades. Their work begins with deciding what a product should withstand and then recreating the conditions it will actually face. A medical device intended for patients cannot be adequately tested only by trained nurses. The model gap shows up with AI agents, too. Testing them on carefully written requests may not tell us how they handle vague instructions, missing information, and unexpected situations in practice.
Safety engineering offers another lesson. Warning people not to do something is weaker than preventing them from doing it. The same applies to agents. Telling an agent not to access a sensitive system is not the same as removing its credentials or blocking network access. This is why I favor enforcing important boundaries in software rather than relying on instructions in a prompt.
The broader lesson is to design systems around how people and agents actually behave, not just how we expect them to behave. Testing needs to account for both the requirements we write down and the conditions the system will encounter.
What I Would Change
- Ask what passed, and under what conditions. A single pass/fail score can hide a lot. This is one reason I’ve advocated for high-dimensional evaluations, which examine specific risks and failure modes rather than relying on one overall score. Even then, ask which assumptions and real-world conditions those evaluations might still miss.
- Review the translation from intent to rules. When AI turns instructions into software or rules for agents, require it to surface added assumptions, changed requirements, and unresolved ambiguities. Spend expert time confirming that the resulting rules capture what was intended.
- Use more realistic evaluations. Combine cheap automated tests with representative workloads, unexpected inputs, and limited deployments. Keep some evaluation cases out of the agent’s development loop so it cannot tailor every solution to the grader.
- Put hard limits outside the model. Use permissions, isolated environments, approval thresholds, and the ability to reverse consequential actions. A prompt telling an agent to be careful is not a security boundary.
- Make failure improve the process. Treat real-world failures as feedback on both the requirements and the tests. Determine whether the implementation was defective, the requirements were incomplete, or the tests assumed unrealistic conditions. Fix the underlying problem rather than just patching the immediate bug.
- Measure the whole cost of confidence. Faster generation is not necessarily better if it creates more review work, testing costs, or production incidents. Measure the full workflow, not just how quickly an agent produces an answer.
The Bottleneck Is Moving
As AI makes it cheaper to produce code, mathematical proofs, and other outputs, the harder work increasingly lies in defining success and establishing confidence in the results. What we’re seeing in coding and mathematics is an early glimpse of what’s coming for other kinds of knowledge work. I expect people to spend less time producing routine work and more time translating human intentions into instructions machines can follow, checking what might have been lost in translation, and evaluating whether the results make sense.
This doesn’t mean humans need to review every step. But we do need to figure out where human judgment matters most, particularly when requirements are ambiguous or mistakes have serious consequences. I would ask every team deploying agents one question: verified against what? If the answer only describes the test, without explaining how it reflects what people intended and what happens in the real world, we still have work to do.
The Looming Debate Over Frontier AI Labs
AI and the Vanishing Apprentice
