Agent demos are easy to build and easy to believe. An agent chains six tool calls, completes a task that would have taken a person twenty minutes, and everyone in the room concludes the hard part is done. The hard part has not started.
The engineering that separates a demo from something you would authorize to touch production is unglamorous. It is three things, and none of them involve the model.
1. Knowing when not to act
An agent that always acts is a liability. The valuable capability is recognizing that confidence is too low to proceed and stopping. That means explicit thresholds, checked deterministically between reasoning steps rather than left to the model's own judgment about its own certainty.
- Confidence gates before any state-changing action
- Policy checks that a step cannot bypass, evaluated in code
- Blast-radius limits — number of records, dollar amounts, irreversibility
- Explicit human approval above configured thresholds
2. Making every side effect reversible
Before an agent gets a tool, that tool needs an idempotency key and a defined rollback path. This is a constraint on your tool layer, not on the agent, and it is the constraint that determines how much autonomy you can safely grant.
In practice this means every tool accepts a request identifier and returns the same result if called twice with it. It means writes go through an interface that records a compensating action. And it means a dry-run mode, because the most valuable thing an agent can do during rollout is tell you what it would have done.
The agent is only ever as safe as its worst tool. Audit the tools, not the prompt.
3. Leaving a trace someone can replay
When an agent does something wrong — and it will — you need to reconstruct exactly why. That means capturing prompts, tool calls, intermediate state, and model versions for every run, and being able to replay that run against a new configuration to verify a fix.
Replay is what turns agent debugging from archaeology into engineering. Without it, every incident review ends in a plausible theory that nobody can test.
Write the authority document first
Before any code, we write down what the agent may do unsupervised, what requires approval, and what it must never attempt. It is usually a single page. It is also the artifact that gets an agent approved for production, because it turns an open-ended capability into a bounded one that a risk function can evaluate.
Teams that skip this end up negotiating scope during the security review, which is the most expensive possible moment to discover you and your stakeholders disagreed about what the agent was for.