Building an agent demo takes an afternoon. Running one in production takes considerably longer, and most of that time goes into things the demo never had to deal with: partial failures, ambiguous inputs, permissions, cost ceilings, and the awkward question of what happens when the model confidently does the wrong thing to a real customer record. The gap between the two isn’t model quality. It’s engineering discipline around a component that is, by design, non-deterministic.
This article covers what actually changes when an agent moves from a notebook to a production system: how to structure the architecture, where to put guardrails, how to evaluate something that doesn’t produce the same output twice, and — importantly — when an agent is the wrong tool entirely.
What Makes an Agent Different From a Chatbot
A chatbot generates text. An agent generates text and takes actions — calling APIs, writing to databases, sending messages, triggering workflows. That single difference changes the risk profile completely. A hallucinated sentence in a chat window is an annoyance. A hallucinated argument to a refund_order call is an incident.
The other structural difference is the loop. An agent runs multiple model invocations in sequence, each conditioned on the results of the last. This means errors compound. A slightly wrong interpretation in step one becomes a confidently wrong action in step four, and by then the model has generated enough intermediate reasoning to make the wrong path look justified. Systems that work fine as single-shot classifiers become unpredictable when you wrap them in a loop.
The Three Things That Actually Break
In practice, production agent failures cluster into three categories:
Tool misuse. The model calls the right tool with wrong arguments, or the right arguments in the wrong context. It passes a customer ID where an order ID belongs, or it calls a write operation during what was supposed to be a read-only investigation.
Loop pathologies. The agent gets stuck retrying a failing call, oscillates between two tools, or keeps refining an answer that was already good. Without hard limits, this burns tokens and wall-clock time with nothing to show for it.
Silent degradation. The agent produces plausible output that is subtly wrong, and nobody notices because there’s no ground truth to compare against. This is the most dangerous category, because it doesn’t page anyone.
Every architectural decision below is aimed at one of these three.
Architecture: Constrain the Blast Radius
The single most useful design principle for production agents is to narrow what the agent is allowed to do until the remaining surface is something you’d be comfortable auditing.
Start With Workflows, Escalate to Agents
Not every problem needs an agent. If the steps are known in advance, write a workflow: a deterministic pipeline where the LLM handles the parts that need language understanding and ordinary code handles everything else. Classification, extraction, summarization, and routing are all better served by a single well-prompted call inside a normal control flow than by a model deciding its own sequence.
Reserve genuine agency — the model choosing which tools to call, in what order, and when to stop — for problems where the path genuinely varies by input. Investigation, triage, and open-ended research fit. “Process this invoice” usually doesn’t. Teams that skip this question end up paying agent-level latency and cost for workflow-level problems.
This is the same instinct behind good solution architecture generally: push determinism as far down the stack as it will go, and reserve flexibility for the layer that actually needs it.
Tool Design Is API Design, With Sharper Edges
The tools you expose define the agent’s capability ceiling and its failure floor. A few things matter more than they do in ordinary API design:
- Make tools narrow and named for intent.
get_open_orders_for_customeris safer thanquery_database. The narrower the tool, the less room there is for the model to compose something you didn’t anticipate. - Validate at the tool boundary, not in the prompt. Prompt instructions like “only use this for read operations” are suggestions. Schema validation and server-side authorization are enforcement. Assume the model will eventually violate every instruction in your system prompt and design so that it doesn’t matter.
- Return errors the model can act on. A tool that returns
500 Internal Server Errorteaches the agent nothing. One that returns “customer_id must be a UUID; received ‘ACME Corp’” lets it self-correct on the next turn. Error messages are part of your prompt engineering surface. - Make destructive operations two-phase. Have the agent propose an action, then require a separate confirmation — from a human, a rules engine, or a second model with a narrower mandate. The cost of this is one extra round trip. The benefit is that irreversible actions never happen on a single model output.
Give the Agent Less Context, Not More
There’s a persistent instinct to stuff everything into context because context windows are large. Resist it. Long contexts degrade instruction-following, increase cost linearly, and make it harder to diagnose why a particular decision was made. Retrieve what’s relevant to the current step, summarize prior turns aggressively, and drop tool outputs once they’ve been used. An agent with a tight, curated context is easier to debug and usually more accurate.
Guardrails: Layered, Not Prompted
Guardrails belong in code, at layers the model cannot reason its way around.
Permission Boundaries
The agent should run with a scoped identity, not a service account with broad access. If the agent is acting on behalf of a user, it should inherit that user’s permissions — an agent helping a support rep should not be able to read records the rep couldn’t open manually. This sounds obvious and is frequently skipped, because wiring per-user credentials through an agent loop is more work than using one shared key.
Budgets and Circuit Breakers
Every agent run needs hard ceilings: maximum turns, maximum tokens, maximum wall-clock time, maximum calls per tool. When a ceiling is hit, the run stops and returns partial results with an explicit failure reason. This is the cheapest possible fix for loop pathologies and it should exist before the first production deployment, not after the first surprising invoice.
Input and Output Filtering
Inputs reaching the agent may contain prompt injection — instructions embedded in a document, a support ticket, or a web page that attempt to redirect the agent. There is no complete defense, which is precisely why the permission boundary above matters: assume injection will sometimes succeed and ensure the resulting actions are still within an acceptable range. On the output side, check for leaked system prompts, PII that shouldn’t be surfaced, and formatting that could be interpreted as markup downstream.
The Human in the Loop, Placed Deliberately
Human review is a guardrail, but it degrades fast if overused — reviewers who approve hundreds of low-stakes suggestions stop reading them. Put humans where the decision is consequential and the review is genuinely tractable: approving a refund above a threshold, confirming an outbound customer message, signing off on a data change. Don’t put them where they’ll rubber-stamp.
Financial services teams have worked through much of this reasoning already; our writeup on agentic AI in banking covers how these controls map onto regulated workflows.
Evaluation: The Part Most Teams Skip
You cannot improve what you can’t measure, and agents are hard to measure because the output space is open-ended and the same input may legitimately produce different valid outputs.
Build a Test Set From Real Traffic
Start by logging everything: inputs, every tool call with arguments and results, intermediate model outputs, final results, and outcomes. From those logs, curate a fixed set of cases with known-good outcomes. This set is your regression suite. It should include the boring successful cases, the known failure modes, and the adversarial inputs you’ve encountered.
The set doesn’t need to be enormous to be useful. A few dozen well-chosen cases that cover your real distribution will catch more regressions than thousands of synthetic ones.
Evaluate Trajectories, Not Just Outputs
For agents, the path matters as much as the destination. Two runs can produce the same final answer while one took three tool calls and the other took fourteen, or one read only permitted data and the other wandered into records it shouldn’t have touched. Assertions worth writing include: did it call the expected tools, did it avoid forbidden ones, did it terminate within budget, and did it ask for clarification when the input was genuinely ambiguous.
Use Model-Graded Evaluation Carefully
Using an LLM to grade another LLM’s output is practical for subjective criteria like tone, completeness, or whether an answer is grounded in the retrieved sources. It works best when the grading rubric is specific and the judge sees a reference answer. It works poorly as a general “is this good?” oracle, and it inherits the judge model’s own biases — including a tendency to prefer longer, more confident answers. Treat model grading as a noisy signal that flags candidates for human review, not as ground truth.
Watch Production, Not Just CI
Offline evaluation catches regressions. It doesn’t catch distribution shift — users asking things you never anticipated, or a downstream API changing its response format. Production monitoring should track tool error rates, turn counts per run, budget-exhaustion frequency, and the rate at which users abandon or correct the agent. Rising average turn count is often the earliest signal that something upstream has changed.
Running all of this reliably needs the same delivery foundations as any other production service — versioned deployments, staged rollouts, fast rollback. If those aren’t in place, an agent will expose the gap quickly, usually during an incident.
Cost and Latency Are Product Constraints
Agent runs are multiplicative: each turn is a full model call including the accumulated context. A ten-turn run with a growing context costs far more than ten times a single call. This has design consequences:
- Route by difficulty. Use a smaller, faster model for routine steps and reserve the largest model for the ones that need it. Many agent loops spend most of their turns on mechanical work that doesn’t need frontier capability.
- Cache aggressively. Stable system prompts and tool definitions are the same on every turn; prompt caching turns that repeated prefix from a per-turn cost into a near-free one.
- Stream and show progress. Users tolerate a slow agent far better when they can see what it’s doing. This is a UI decision that materially affects whether the feature gets used.
When Not to Build an Agent
Skip the agent if the task is deterministic, if the cost of a wrong action is high and can’t be gated by review, if you have no way to evaluate correctness, or if latency requirements are tight. Skip it if a well-tested rules engine already handles the case — deterministic automation is cheaper, faster, and auditable, and traditional RPA and intelligent automation remains the better answer for high-volume, well-specified processes. The interesting systems are usually hybrids: rules where rules work, models where judgment is required, and a clear boundary between them.
Conclusion
Agents earn their keep on problems where the path varies and judgment is required, and they cost more than they’re worth everywhere else. The teams that ship them successfully aren’t the ones with the best prompts — they’re the ones who treated the model as an untrusted component in an otherwise well-engineered system: scoped permissions, validated tool boundaries, hard budgets, curated evaluation sets, and honest production monitoring. Get those in place first, and the model becomes something you can iterate on safely. Skip them, and every model upgrade is a coin flip.
