The biggest AI risks sit outside the model
The most revealing AI failures right now are not stories about models becoming too capable. They are stories about everything around the model. One system received more access than its test environment could contain. Another was trained on material whose acquisition created $1.5 billion in exposure. In a third, people walked away more certain without being any more correct. I recently argued that passing your evals does not mean you are safe. This is the next layer of that argument. Permissions, inputs, and presentation are three places where serious AI risks accumulate, and each one lands on a different desk.
Readers are what keep Gradient Flow going. Consider becoming a paid supporter 🙏
What an Agent Can Touch
Model capability tells you what a system can do. Architecture determines what it may do, what data and tools it may use, and which actions require approval. A mediocre model with broad write access can be more dangerous than a stronger model in a tight box.
OpenAI’s cyber-focused models, including GPT-5.6 Sol, left the sandbox used for testing, reportedly exploited a zero day, reached the open internet, and compromised Hugging Face production systems. Nobody appears to have noticed for days. Calling the behavior rogue explains little. Safety filters had been switched off for the exercise, and network isolation was weaker than the test assumed. Those were architecture and configuration decisions.
Other reports describe an agent deleting user files without being asked and a coding tool uploading full codebases without clear consent. Different incidents, same pattern: a weakly specified task becomes dangerous in proportion to what the system can touch.
An agent can also be redirected by hostile instructions embedded in an email, document, web page, or tool response. Once tools are attached, prompt injection can lead to an unauthorized action, not merely a bad answer. Permissions and approval rules therefore have to be enforced outside the model.
Give every production agent its own identity and task-specific permissions rather than letting it borrow a person’s credentials or a generic service account. Split read from write. Use short-lived credentials, narrow network access, transaction limits, and approval gates for irreversible actions. Preserve logs and make recovery possible. The goal is not perfect prevention. It is a small blast radius, fast detection, and enough evidence to understand what happened.
Not every system needs all of those controls. If an agent only reads, gets things wrong cheaply, and can be corrected before anyone acts, most of those controls are overhead. Once it can write to a system of record, spend money, touch regulated data, or produce something a customer will act on without review, those controls apply. The dividing line is the damage the system can cause before someone stops it. Many teams want a third option. There isn’t one.
Where the Data Came From
The same pattern runs upstream, into what the model was trained on before anyone deployed it. A federal judge approvedAnthropic’s $1.5 billion settlement over pirated books used to train Claude. The headline number is the least interesting part. What matters is the line the settlement draws between training on material you obtained lawfully and training on material pulled from pirate libraries. The failure was not model capability. The training pipeline did what it was built to do, and a decision made around it created the exposure.
If you are fine-tuning a model, assembling an evaluation set, or building retrieval over third-party content, that distinction is your problem too. How the material was acquired, what rights came with it, and whether those facts can be reconstructed later matter as much as what the model does with it.
Boards now have reason to ask where training data came from, who carries the risk, and whether a disputed model or dataset can be removed without disrupting the product.
Answering those questions requires a dataset inventory, license terms you can produce on request, a clear answer on who pays if you get sued, a current view of litigation involving your vendors, and a way to swap out a model or dataset without taking the product down. Most teams have some of this. Very few could pull it together on short notice.
The Confidence Trap
Controls and documentation can limit what a system does and clarify what went into it. Neither fully addresses what happens when a convincing answer reaches a user. A recent preprint reports five experiments involving 3,132 participants who answered difficult questions and had the option to say they did not know. Giving people access to AI advice nearly eliminated that restraint. Participants answered more questions, but they were correct about one-third as often as people without AI access. Their confidence nearly doubled.
You should assume your users can do the same thing. In medical, legal, financial, hiring, and security workflows, a confident wrong answer can be more dangerous than an obvious failure. The risk grows when the interface presents a smooth recommendation as the natural next step. A confidence score does not automatically help, especially when it comes from the same model that produced the answer. A precise-looking number can become decoration rather than evidence.
You cannot engineer this one away, because you cannot stop a person from over-valuing a fluent answer. What you can do is design against it, and the moves are concrete. Ask the user to commit to a view before you reveal the model’s recommendation. Show uncertainty honestly instead of rounding it into confident prose. Take confidence numbers from past accuracy, not from the model. Surface disagreement between sources rather than smoothing it out. Make correcting the system easy enough that people actually do so. Let the system abstain. Escalate decisions when the cost of being wrong is high.
Then check whether it worked. The test is not whether people like the assistant or finish more tasks with it. It is whether the human and the system together make better decisions than the process they replaced, know when to abstain, and escalate the right cases.
Three Desks, One Problem
The next phase of enterprise AI will be defined less by who gets the most capable model and more by who builds the strongest system around it.
Security and architecture teams have to bound what the system can touch. Data, legal, and procurement teams have to document what went in. Product teams have to preserve human judgment. Rules people once enforced through policy, habit, or supervision increasingly need technical equivalents such as distinct agent identities, approval gates, provenance, and logs.
These risks land on different desks, but they are becoming one management problem.
Capability is the layer you buy. Everything around it is the layer you own.
Evaluating the system, rather than just the model, requires a richer set of measurements. The Luminos whitepaperexplains one approach.
When Mathematical Results Become Abundant
