Nine Practical Rules for Agents Doing Real Work
In recent conversations with crews building agents, I keep hearing the same lessons. Teams with very different products are arriving independently at almost the same architectural choices. That convergence feels important. In recent posts, I argued that passing your evals does not mean an AI system is safe, and that many of the most consequential risks sit outside the model itself. This is the constructive follow-up: once an agent is doing real work, what should you actually build around it? I think a practical architecture playbook is beginning to emerge.
Gradient Flow exists because readers pitch in. Consider becoming a paid supporter 🙏
1. Put hard constraints in software, not prompts
Models are good at flexible reasoning. Ordinary software is good at state, permissions, calculations, retries, and predictable control flow. Reliable systems keep that division explicit rather than asking a prompt to guarantee something important. A prompt is guidance. It stays negotiable no matter how firmly you word it, which is fine for judgment calls and a real problem for anything that can’t be wrong.
The dividing line is simple: let the model handle ambiguity, which is where it creates value. Put calculations in trusted code, enforce permissions through policy systems, validate code with compilers and tests, and check important factual claims against authoritative sources. Ask what the model can currently violate that ordinary software could prevent. When a failure would create a material consequence, the architecture should make that violation impossible or require explicit approval.
2. Give the agent only as much autonomy as the job needs
One pattern I keep seeing is teams giving agents more autonomy than the job requires. That excess autonomy creates more possible paths, more opportunities for error, higher operating costs, and a harder governance problem. For many applications, conventional software should control the workflow while the model handles the smaller number of steps that genuinely require judgment. Greater autonomy can be justified when the task requires exploration, open-ended planning, or creating new tools.
I would treat autonomy as a deliberate design choice rather than a default. Start with a flexible workflow, observe which paths repeat reliably, and turn those stable paths into ordinary code. List every point where the agent chooses its next action, then ask which of those choices the task truly requires. A mature agent faces fewer open-ended choices over time, not more.
3. Build the agent around the domain’s trusted process
A support workflow already specifies how to triage a request, what information to collect, when to escalate, and which actions require approval. A medical workflow does the same through diagnostic protocols. When a domain already has a proven checklist, protocol, or decision process, use that as the agent’s structure rather than starting with a generic loop that tells the model to make a plan and keep acting until it is done.
This makes the system more reliable because the structure has already been tested by the field. It also makes the system easier to understand for the people who have to trust it and sign off on it. The best agent architecture often looks less like a general-purpose digital employee and more like the field’s existing best practice made executable.
4. Design for recovery, not a flawless first pass
Small errors compound quickly in long workflows. A system that succeeds 95 percent of the time at each step has only about a 60 percent chance of completing ten independent steps without an error. That helps explain why a three-step demo can look extraordinary while a longer business process falls apart.
Do not build around the assumption that the model will stop making mistakes. Add checkpoints, checks that confirm each action produced the intended result, retries, reversible actions, and the ability to resume from a known-good state. Measure recovery separately from first-attempt accuracy. A system that detects drift and corrects itself is more useful than one that looks perfect until the first unexpected tool response.
5. Evaluate the model and harness as one system
What exactly are you evaluating when you test an agent? The harness is the software surrounding the model, including its tools, context management, memory, policies, and recovery logic. That surrounding system can affect performance as much as changing the model. In one comparison, the same open model showed an 18 percentage point difference between the best and worst harness configurations. That is one of the findings I would take most seriously when comparing models.
A leaderboard is only a starting point. Evaluate the model with your tools, context, permissions, and failure modes attached. Then rerun the evaluation whenever either the model or the harness changes. Changing either half means you are evaluating a new system.
6. Keep multi-agent teams small, and let someone disagree
Multi-agent systems can look sophisticated while mostly adding coordination overhead. The more effective pattern is usually a clear orchestrator with a small number of specialists, each with a distinct role, tool set, permission scope, and only the information it needs. Larger teams can help when the work is genuinely parallel or when independent perspectives are part of the requirement, but large swarms often create duplicated work and make failures harder to trace. One role deserves special protection: a critic or breaker with explicit criteria and the authority to block or escalate. Knowing when not to answer is something the system must be designed to do, not merely a personality you request in a prompt.
7. A bigger toolbox can make the agent worse
In one test, a larger tool set was slower and slightly less accurate even though it contained every tool in the smaller set. This may be the most counterintuitive rule on the list. The extra tools were not bad. The problem was overlap. When several tools can plausibly handle the same request, the model spends attention choosing among near-duplicates instead of doing the work.
More tools also give the agent more systems it can read from or write to, and they create more sequences of tool calls for the team to test. Favor a compact toolbox in which every tool has a clearly different job. Run required setup tools automatically, and log which tools get picked, what goes in and out, and how often they fail. If two tools could plausibly handle the same request, combine them, route between them in software, or remove one.
8. Context, memory, and enterprise knowledge are different systems
The distinction is operational: context is the information the model needs during the current run, memory is what the system should carry forward from earlier work, and enterprise knowledge is the governed collection of documents, records, and policies it may consult. Treating them as one undifferentiated store creates both quality and security problems. They can share infrastructure, but they should not share the same retention, retrieval, and access rules by default.
Memory in particular should preserve lessons, not transcripts. What matters is the mistake to avoid, the technique worth reusing, or the preference that should change future behavior. The test is not how much the system remembers. Inspect what it saved, what it later retrieved, and whether that information changed the result.
9. Fix the knowledge before you pay for a bigger model
When an agent performs poorly, the default reaction is often to buy a stronger model or begin fine-tuning. Before changing the model, I would look closely at what it is being given to work with. One team replaced a pile of raw support documents with a diagnostic playbook. A router translated a customer’s complaint into a symptom, then a problem type, then the relevant handbook. Tokens fell 43 percent and errors fell 48 percent without changing the model.
Retrieval is not a feature you install once and forget. It fails when a user’s words do not match the document’s words, when information is buried in tables or PDFs, or when sources conflict. Before changing the model, improve the structure, routing, and governance of the knowledge it receives. The model is the component you can buy. The system around it is what you own.
If agents and real-world AI deployment are on your roadmap, come spend two days with the teams already doing it. AI Conference, San Francisco, Sep 30–Oct 1. 20% off with gradientflow20.
The Case for AI Data Centers Nobody Is Making Well
The Memory Bottleneck in Frontier AI
