Most AI agent architectures are designed for capability and then retrofitted for control, which is why so few reach production inside a real company. Here is what changes when an agent has to pass an audit — permissions, data boundaries, cost governance, human approval, and the evidence trail — and how to build for it from the start.
There is a stage in enterprise AI adoption that almost nobody writes about, because it is not fun.
The pilot worked. An agent was wired to some internal systems, it answered questions or filed tickets or reconciled records, and the demo went well enough that someone senior asked what it would take to roll it out properly.
Then the questions start. Who authorised it to touch that system? What data left the building? What happens if it does the wrong thing at 3am? Can you show me every action it took last Tuesday? What is the monthly cost, and what stops it doubling? Which regulation covers this?
Most agent architectures have no answer to any of that, because they were designed to be capable and control was assumed to be a wrapper you add later. It is not a wrapper. It is a structural property, and retrofitting it usually means rebuilding.
This post is about what an agent architecture looks like when those questions are load-bearing from the start.
TL;DR — key takeaways
- The agent is not the unit of permission. Give each tool its own identity and scope; an agent that inherits one broad service account is an audit failure regardless of how well it behaves. - Act on behalf of the user, not as a superuser. The agent should never be able to read or change something the requesting person could not. - Budgets are architecture. Iteration caps, token ceilings, per-tenant cost limits and wall-clock timeouts belong in the orchestrator, not in a prompt. - Separate reversible from irreversible actions. Reversible actions can be autonomous; irreversible ones need approval, and the boundary should be declared in the tool definition. - Every run needs a replayable trace. Inputs, tool calls, arguments, results, token counts, and the termination reason — enough to reconstruct what happened without the original session. - Data boundaries are a deployment decision. Which model, hosted where, seeing what, determines whether the system is legal before it determines whether it is good. - Evaluate the guardrails, not just the model. Deterministic controls fail silently and are the least-tested part of most systems.
Why enterprise agents fail differently
A consumer agent that misbehaves produces a bad experience. An enterprise agent that misbehaves produces an incident with a regulatory dimension.
The failure modes that matter change accordingly:
| Consumer failure | Enterprise equivalent | Why it is worse | |---|---|---| | Wrong answer | Wrong answer acted on by a system of record | It propagates; downstream processes consume it | | Infinite loop | Runaway spend against a shared budget | Finance notices before engineering does | | Hallucinated citation | Fabricated figure in a report sent to a client | Reputational and sometimes contractual exposure | | Leaked chat history | Cross-tenant or cross-department data exposure | Reportable breach | | Tool misuse | Unauthorised write to a production system | Change-control violation |
Notice that none of these are model-quality problems. A better model reduces the frequency of the first row and does nothing whatsoever about the rest. That is the core insight: enterprise readiness is almost entirely a systems problem, and the model is the component you have the least need to change.
I have written about the general shape of this in what actually breaks when agents reach production. This post is the enterprise-specific layer on top.
Permission belongs to the tool, not the agent
The most common architectural mistake is giving the agent one service account with the union of every permission any of its tools might need.
It is convenient, it makes the pilot work, and it is indefensible in review. The agent can now do anything any of its capabilities allow, in any combination, regardless of context. There is no way to answer "what is this agent allowed to do" other than reading the whole permission set and imagining the worst case.
The alternative: each tool carries its own identity and its own scope. The CRM read tool has read access to the CRM and nothing else. The ticket creation tool can create tickets and cannot close them. Scopes are declared alongside the tool definition, so the permission surface is a document you can hand to a reviewer.
This costs more to set up and it makes the blast radius of a compromised or misled agent bounded and describable, which is the entire point.
Act on behalf of the user
The second structural rule: an agent invoked by a person should operate with that person's authority, not its own.
If a sales representative asks the agent about an account they cannot access, the correct outcome is that the agent cannot access it either. If the agent runs with a broad service identity, it will happily retrieve and summarise data the requester was never entitled to see — and it will do so in a way that leaves no trace resembling a permission violation, because from the system's perspective nothing was violated.
This is the mechanism behind a whole category of quiet enterprise data leaks: not exfiltration, just an over-privileged intermediary being helpful.
Implementing it means propagating the caller's identity through the orchestrator into every tool call, and having tools enforce authorisation themselves rather than trusting the agent layer. It also means accepting that the agent will sometimes have to say "you do not have access to that", which is the correct behaviour and will nonetheless generate complaints.
Reversible and irreversible actions are different categories
Autonomy is not a single dial. It is a property of individual actions.
Reading a record, running a query, drafting a document, computing a summary — these are reversible. If the agent does them wrongly, the cost is wasted compute and a bad draft. Let it do them freely; requiring approval for every read makes the system useless.
Sending an email to a customer, writing to a system of record, issuing a refund, deleting anything, deploying anything — these are irreversible or expensive to reverse. The agent should prepare them and a human should confirm them.
Make this a declared property of each tool rather than a judgement the model makes at runtime:
| Tool property | Meaning | Runtime behaviour | |---|---|---| | reversible | No lasting external effect | Execute autonomously | | confirm | Effect is external but recoverable | Execute after explicit user confirmation | | dualcontrol | High-value or high-risk | Requires a second human approver | | forbiddenincontext | Not permitted for this tenant or workflow | Never surfaced to the model at all |
That last row matters more than it looks. The safest way to prevent an agent from taking an action is to not offer it the tool. Filtering the tool list per context is cheaper and more reliable than any prompt instruction telling the model to refrain.
Budgets are part of the architecture
An agent without budgets is an outage waiting for the right input.
Four separate limits, all enforced by the orchestrator rather than requested in a prompt:
Iteration cap. Maximum loop turns before the run terminates and reports what it achieved. Nearly every runaway-cost story is an agent that could not tell an action had failed and retried indefinitely.
Token ceiling per run. Independent of iterations, because a single step over a large context can be expensive on its own.
Wall-clock timeout. Some tool calls hang. A run holding resources for an hour is a availability problem even if it costs nothing.
Cost budget per tenant per period. The one finance cares about. Without it, one department's enthusiastic adoption becomes an unbudgeted line item that arrives a month late.
When a budget is hit, the agent should terminate gracefully and report partial progress with the reason. Silent truncation is worse than failure, because the caller cannot distinguish "finished" from "gave up".
The data boundary is decided before anything else
Which model, running where, seeing what, is not an optimisation question. It determines whether the system is permissible at all, and it constrains everything downstream.
The practical options, roughly in order of restriction:
Self-hosted open-weight model on your own infrastructure. Nothing leaves. Highest control, highest operational burden, and you now own a GPU capacity planning problem — I have written a guide to choosing that hardware.
Managed model in your own cloud tenancy, within your compliance boundary. A common middle ground for regulated industries.
Vendor API with contractual guarantees on retention and training use. Adequate for many enterprises, unacceptable in some jurisdictions and sectors.
Vendor API with default terms. Fine for public data, and the thing that gets pilots quietly shut down when someone reads the terms.
Decide this first, because it dictates model choice, which dictates capability, which dictates what the agent can realistically do. A team that designs an agent around a frontier model and then discovers the data cannot leave the building has to start over.
Related: retrieval expands the boundary. An agent that queries an internal vector store is sending fragments of internal documents to whatever model generates the answer. If the documents are sensitive, the boundary applies to those fragments, not just to the user's question.
Untrusted input is everything the agent did not write
Prompt injection in an enterprise context is not a curiosity. It is the mechanism by which an agent with legitimate permissions is convinced to use them wrongly.
Every piece of text entering the context that the agent did not generate is untrusted: retrieved documents, ticket bodies, email content, web pages, file uploads, API responses, and — importantly — content written by other users of the same system.
The defences that actually work are structural rather than instructional:
- Filter before the model. Injection filtering runs ahead of every LLM call, not as an output check. This is the same discipline I applied in the clinical assistant, where a prompt-injection filter sits ahead of every model call by construction. - Keep permissions minimal. An injected instruction can only cause harm through a tool the agent actually has. Scope is the real defence. - Confirm irreversible actions. A human in the loop breaks the chain from injected instruction to external effect. - Separate instruction and data channels where the model supports it, so retrieved content is structurally marked as content rather than command.
Telling the model in its system prompt to ignore instructions in retrieved text is not a defence. It is a preference, and it can be argued with.
The evidence trail
"Show me what it did last Tuesday" is a routine audit request and most agent deployments cannot answer it.
What a sufficient trace contains, per run:
| Field | Why it is needed | |---|---| | Trace ID | Correlates everything below across services | | Requesting identity | Who invoked it, under whose authority | | Full model inputs and outputs per step | The only way to reconstruct a decision | | Every tool call: name, arguments, result, duration | What actually touched other systems | | Token counts and cost per step | Attribution and anomaly detection | | Termination reason | Completed, budget exceeded, error, user cancelled | | Approval events | Who confirmed what, and when |
Two constraints on this data that are easy to miss. It contains whatever the agent processed, so it inherits the sensitivity of that content and needs the same retention and access controls as the source systems. And it needs to be queryable by a human under time pressure — logs that exist but cannot be searched during an incident are compliance theatre.
The test is simple: can you replay a failed run from storage, without the original session, and see why it did what it did? If not, you will be debugging production agent failures by guessing, and agent failures are rarely reproducible on demand.
Evaluate the guardrails, not just the model
Teams build evaluation harnesses for the probabilistic parts of the system and code-review the deterministic parts, because deterministic code is simple and simple code is assumed correct.
Simplicity guarantees the code does what it says. It says nothing about whether what it says is right.
I learned this expensively: a rule-based emergency-detection gate in a clinical assistant was perfectly deterministic and missed half its cases, because it ran before text normalisation and matched clinical vocabulary against lay phrasing. Stratified evaluation caught it; code review had not, and neither had unit tests written from the same assumption as the implementation.
For an enterprise agent, that means building evaluation cases for the control layer specifically:
- Does the permission check actually deny a request the caller is not entitled to make, with realistic phrasing? - Does the iteration cap fire and terminate cleanly, and does the caller get a useful partial result? - Does an injected instruction inside a retrieved document reach a tool call? - Does an irreversible action taken through an unusual path still require confirmation? - Does the cost budget hold under a pathological input?
Each of these is a test with an expected outcome, and each should run in CI with a hard threshold. Guardrails that are never tested against adversarial input are decoration.
What to build first
A pragmatic order, for a team going from pilot to production:
1. Fix the identity model. Per-tool scopes, caller identity propagation. Everything else is unreviewable until this exists. 2. Add budgets. Cheapest large risk reduction available. 3. Classify tools by reversibility and gate accordingly. 4. Build the trace. Before you need it, because you will need it during an incident. 5. Add injection filtering ahead of every model call. 6. Write control-layer evaluations and put thresholds in CI. 7. Then improve capability — better retrieval, better prompts, a stronger model.
Almost every team does this list backwards, and that is why pilots stall in review rather than in engineering.
Closing
The question that decides whether an enterprise agent ships is not whether it completes the task. Pilots already proved that.
It is:
"When it does the wrong thing — and it will — what is the largest possible consequence, who authorised that scope, and can you show me exactly what happened?"
An architecture that can answer those three clauses will get deployed even if its capabilities are modest. One that cannot will stay a pilot forever, no matter how impressive the demo was.
Capability is the part that gets the meeting. Control is the part that gets the rollout.
If you are taking an agent from pilot to production and want a second pair of eyes on the architecture, get in touch.
Frequently asked questions
What makes an enterprise AI agent different from a normal AI agent?
The controls, not the capabilities. An enterprise agent has to operate within a permission model tied to real identities, keep data inside a defined compliance boundary, enforce spending limits, distinguish reversible from irreversible actions, and produce an audit trail somebody can query during an incident. None of that is about model quality, which is why upgrading the model rarely moves a stalled enterprise deployment.
How do you control what an AI agent is allowed to do?
Scope each tool individually rather than giving the agent one broad service account, propagate the requesting user's identity so the agent can never exceed that person's own access, filter the tool list by context so forbidden actions are never offered to the model at all, and gate irreversible actions behind human confirmation. Prompt instructions telling the model to behave are a preference, not a control.
How do you stop an AI agent from spending too much money?
Enforce four independent limits in the orchestrator: an iteration cap, a token ceiling per run, a wall-clock timeout, and a cost budget per tenant per period. Runaway spend almost always comes from an agent that cannot tell an action failed and retries indefinitely, so iteration caps and repeated-call detection matter more than the model's per-token price. When a limit is hit, terminate cleanly and report partial progress with the reason.
Should enterprises self-host LLMs or use vendor APIs?
It depends on where your data is permitted to go, and that question should be settled before any architecture work. Self-hosting an open-weight model keeps everything internal at the cost of owning GPU capacity planning. A managed model inside your own cloud tenancy is a common middle ground for regulated sectors. A vendor API with contractual retention and training guarantees is adequate for many enterprises. Default vendor terms are fine for public data and are what typically gets pilots shut down in review.
What is prompt injection and why is it worse in enterprise systems?
Prompt injection is instructions hidden in content the agent processes — a document, a ticket, an email, an API response — that redirect its behaviour. It is worse in enterprise settings because the agent holds real permissions against real systems, so a successful injection converts legitimate access into an unauthorised action. The effective defences are structural: filter before the model, keep tool scopes minimal, require confirmation for irreversible actions, and separate instruction from data channels.
What should you log for AI agent auditing and compliance?
Per run: a trace ID, the requesting identity, every model input and output, every tool call with arguments and results, token counts and cost per step, the termination reason, and any approval events. The trace inherits the sensitivity of whatever the agent processed, so it needs the same access controls as the source systems. The working test is whether you can replay a failed run from storage without the original session and see why it behaved as it did.
How do you evaluate AI agent guardrails?
Write test cases for the control layer specifically and run them in CI with hard thresholds: does a permission check deny an unentitled request under realistic phrasing, does the iteration cap terminate cleanly with a useful partial result, does an injected instruction inside a retrieved document reach a tool call, does an irreversible action still require confirmation via an unusual path. Deterministic guardrails are auditable, which teams often mistake for verified — they fail silently and are typically the least-tested part of the system.
Related reading
- AI Agents from Prototype to Production: What Actually Breaks — the engineering failures underneath the governance layer. - From 50% to 100% Emergency Recall: Debugging Safety Routing in a Clinical RAG Assistant — why deterministic guardrails need evaluation too. - What GPU Should a Company Buy for Internal AI Infrastructure? — the hardware decision behind self-hosting.