Building AI Agents Is Getting Easier. Running Them Isn’t.
In August, I wrote about teams building very different AI products arriving at similar architectural choices. I distilled my research and conversations into nine practical rules. I’ve since been looking at deployments in banks, utilities, manufacturers, retailers, and other organizations outside tech. Do those rules hold up when agents are doing real work?
For the most part, they do. But these examples suggest something else. Getting agents into production often depends on work cleaning up data, documenting processes, and improving internal software systems. The agents are benefiting from investments made well before the current AI boom. Most of the published results come from vendors or the companies themselves, so the numbers deserve some skepticism.
1. Limit What Agents Can Do, Not Necessarily How They Reason
Production agents generally operate within clear boundaries, especially when mistakes carry consequences. At Duke Energy, agents coordinate engineering simulations used for electrical grid planning, while engineers review the results and make the decisions. DBS Bank uses chains of 70 to 80 specialized agents to help prepare credit memos, with people responsible for the final analysis.
This reinforces two earlier rules: put hard constraints in software and give agents only as much autonomy as the job requires. But in a recent conversation with Nick Ung and Akshay Sharma at Lyft, Nick raised an important qualification. Overly scripted workflows can prevent agents from handling situations that developers didn’t anticipate. Lyft routes sensitive issues involving fraud or safety to humans without prescribing every step an agent must take. The distinction is between limiting an agent’s authority and unnecessarily restricting how it reasons through a problem.
2. The Business Systems Matter More Than the Chatbot
The more interesting deployments connect agents to data and tools that companies already rely on. Trane Technologies connected an agent to live HVAC equipment data, allowing technicians to investigate problems through natural-language questions. An internal benchmark reduced a 20-minute diagnostic workflow to about 20 seconds. The model didn’t replace the underlying engineering systems. It made their capabilities easier to use.
Consider HEMA, a Dutch retailer whose assistant builds on years of work maintaining a service catalog connecting teams, systems, and APIs. Tottenham Hotspur similarly spent roughly two years consolidating customer records and replacing fragmented systems before launching its fan-support agent.
In an earlier conversation with Andrew Moore, we discussed how AI can help automate this groundwork, including the difficult task of matching records across different systems. His team at Lovelace reports compressing some data preparation projects from months to days. This adds a wrinkle to my earlier rule to fix the knowledge before paying for a bigger model. The work still matters, but AI can increasingly help do it.
3. Companies Are Building Agent Infrastructure, Not Just Agents
This is one development I didn’t emphasize in August. As more teams start building agents, it makes less sense for each application to have its own access controls, monitoring, evaluation tools, and cost management. Companies increasingly need infrastructure that multiple teams can share.
Wood Mackenzie found that 88% of its AI proofs of concept had never reached widespread deployment. Its APEX platform gives teams common infrastructure for developing and operating agents. Brazil’s Itaú created IARA to govern model access, usage costs, and AI policies. At MRH Trowe, approximately 400 employees gained access to a governed agent platform in its first production month, at a reported cost of roughly $14 per user.
Lyft offers another example. Its team treats evaluation as shared infrastructure, combining production records of agent behavior with offline tests and automated assessments. Building an individual agent is getting easier, but keeping a growing collection of agents reliable requires a different set of capabilities.
4. Domain Experts Are Becoming Agent Builders, but Engineers Still Own the Platform
At Grand Isle Shipyard, an operations executive without an IT background built about 20 agents for fleet safety and compliance. Agricultural equipment manufacturer AGCO has roughly 2,000 employees in its AI maker community and several hundred agents in production. People who understand the work are increasingly helping to automate it.
Lyft provides a useful example of how responsibilities are divided. Domain experts who review customer-support tickets write prompts for agents handling relatively simple customer issues. Engineers provide the tools and safeguards, including automated checks for mistakes such as invalid tool references. Engineers also review prompts before deployment, while data scientists run evaluations. More complex agent workflows remain with technical teams.
I think this division of labor makes sense. Business experts understand the problems and how they should be resolved, while engineers make sure the agents can operate reliably. But it raises another question: who will maintain, test, and eventually retire hundreds or thousands of employee-built agents?
5. Be Careful About What Agent Success Metrics Actually Measure
Reported success rates can be difficult to compare because companies measure different things. Tottenham Hotspur reports 86% engagement with its fan-support agent, but only 20% of interactions are resolved without human help. The TSA reports 96% resolution, but only for routine inquiries. A new anti-money-laundering agent at Amalgamated Bank promises time savings exceeding 90%, but those figures are projections.
These numbers tell us something, but they often leave out errors, correction costs, and how much of the complete process is automated. This brings me back to my earlier argument that evaluation must cover both the model and the entire agent system. At Lyft, the team examines an agent’s trajectory, the sequence of questions, tool calls, and decisions it makes, rather than just the final response. An agent might reach the right answer after consulting the wrong database or making a customer repeat information. Looking only at the outcome would miss those problems.
When evaluating claims about production agents, I would ask:
- What exactly is being measured, and what is the denominator?
- How often does the agent make mistakes or require human intervention?
- Who measured the results, and are they actual or projected?
- How much of the end-to-end process does the agent handle?
Taken together, these cases suggest that getting agents into production is as much a test of a company’s existing systems as its AI capabilities. Agent development is becoming both a shared engineering responsibility and something domain experts can participate in. Companies that have invested in reliable data, well-documented processes, and accessible software systems have a head start.
There’s another challenge approaching. JPMorgan is restricting outside agents from accessing customer accounts, while HSBC is opening a controlled channel for corporate clients’ AI tools. Companies will increasingly need to govern not only their own agents, but agents arriving from outside. The same investments in data quality, permissions, and reliable interfaces should become even more valuable.

