When Agents Go Wrong — Agents at Work

An agent is not just a chatbot with better manners. It is software that can pursue a goal, choose steps, use tools, and take action across systems. That shift changes the risk equation. A bad answer can be corrected; a bad action can freeze an account, disclose a file, order the wrong inventory, deny a benefit, or create a chain of evidence no one can later explain. The strategic question is not whether agents are safe or unsafe. It is where delegated action creates enough advantage to justify new forms of control.

The bet is delegated action, not smarter chat

The bet is that work can move from human-operated workflows to software-led execution. The stake is operating leverage: faster handling of claims, fewer missed fraud signals, better machine uptime, sharper markdowns, lower caseload backlogs. The same shift also changes accountability. When a system drafts an answer, judgment remains visibly human. When a system calls tools, moves data, and triggers decisions, accountability becomes distributed across the model, the data, the workflow, the policy, and the people who approved the design.

Agents go wrong because they combine uncertainty with agency. Traditional software tends to fail inside predefined paths. Agents can choose a path. That is their value and their risk. A hospital readmission agent may prioritize outreach to patients most likely to return; a bank fraud agent may decide which cases need review; a factory maintenance agent may open a service order before a line fails. Each example creates value only if the system can act, but each action changes the cost of error.

The first decision is therefore not whether to use agents. It is where action should be delegated, where advice is enough, and where human judgment remains the economic bottleneck worth preserving.

⚠️

The control point moved. The critical risk is no longer only what the model says; it is what the system is allowed to do after the model decides.

The goal can be wrong even when the answer is right

The bet is that agents can optimize outcomes at scale. The stake is that the chosen outcome may be too narrow, stale, or politically convenient. A system can perform exactly as specified and still damage the enterprise if the target is poorly chosen. A retailer that asks an agent to reduce aged inventory may protect margins in one region while eroding loyalty in another. A public agency that optimizes caseload speed may improve throughput while pushing complex cases into a harder appeals queue.

This is not a model-quality problem in the narrow sense. It is an objective-design problem. Agents need a goal, and goals translate strategy into machine-readable choices. If the goal rewards speed, the system will find speed. If it rewards cost reduction, it will find cost reduction. If the broader institution depends on trust, fairness, resilience, or regulatory defensibility, those constraints must be real operating limits rather than after-the-fact values statements.

The decision to weigh is whether the metric being automated is a faithful proxy for the outcome actually valued. The warning sign is not simply a surprising error; it is a pattern of locally rational decisions that accumulate into strategic harm.

A bank fraud agent that minimizes false negatives may create unacceptable false positives and customer attrition.
A hospital readmission agent that optimizes bed utilization may underweight social factors that make follow-up harder.
A factory maintenance agent that minimizes downtime may over-order parts and mask supplier fragility.
A retailer pricing agent that protects weekly margin may train customers to wait for deeper markdowns.

The autonomy boundary is the real design choice

The bet is that autonomy can remove friction from high-volume decisions. The stake is that too little autonomy produces theatrical innovation, while too much creates invisible risk. The useful question is not whether there is a human in the loop. It is where human judgment changes the outcome enough to justify the delay, cost, and accountability burden.

Autonomy is not binary. An agent can recommend, draft, prepare, execute within limits, or execute and escalate exceptions. A bank may allow an agent to assemble a fraud case but require human approval to freeze an account. A manufacturer may allow automatic creation of a work order under a cost threshold but require review for production-line changes. A hospital may let an agent schedule a follow-up call but not alter a care plan.

The boundary should follow reversibility, materiality, and public trust. Low-value, reversible actions can tolerate more automation. High-impact, hard-to-reverse actions need stronger gates. The decision is where delay is a control, where delay is waste, and where the organization is confusing reassurance with real risk reduction.

Autonomy is not binary; agents can recommend, draft, prepare, execute within limits, or execute and escalate.

Context failures become governance failures

The bet is that agents can use the right context at the right moment. The stake is that context is often the least governed part of the system: old policies, incomplete customer histories, mislabeled documents, legacy permissions, or informal exceptions that everyone knows but no database captures. When an agent acts on partial context, the failure can look intelligent, confident, and procedurally clean.

The most dangerous failures are not always dramatic hallucinations. They are plausible decisions made from the wrong evidence. A public agency agent may prioritize a case using outdated eligibility rules. A claims agent may summarize a file but miss a document stored outside the normal repository. A sales-support agent may propose terms based on a historical exception that should never be repeated. In each case, the model is not the only issue; the enterprise memory is fragmented.

Prompt injection raises the stakes because the context itself can become hostile. If an agent reads untrusted emails, documents, tickets, or web pages, malicious instructions can be smuggled into material the agent treats as evidence. The decision to watch is whether the system separates instructions from information, trusted sources from untrusted ones, and policy from persuasion.

⚠️

Plausible is not governed. A clean explanation is not proof that the agent saw the right data, applied the right policy, or had the right authority.

The most dangerous failures are confident decisions made from the wrong evidence.

Tool use turns small errors into balance-sheet events

The bet is that agents can connect judgment to execution through tool invocation. The stake is that every connected tool gives the agent a way to create real-world consequences. A wrong summary is a quality issue. A wrong payment instruction, shipment release, access change, account hold, or procurement request is an operating event.

The risk rises with side effects. Reading a document is not the same as changing a record. Drafting a response is not the same as sending it. Recommending a discount is not the same as applying it across an e-commerce channel. The more an agent can change state in core systems, the more controls must resemble financial controls rather than content moderation.

The strategic measure is blast radius. How much money can move? How many customers can be affected? How many records can be changed? How visible is the error, how reversible is it, and how quickly would it be detected? The decision is not only which tools an agent can use, but what level of consequence is acceptable before stronger approval, limits, logging, or rollback are required.

As agents change state in core systems, controls must resemble financial controls, not content moderation.

Assurance has to become continuous

The bet is that agents can improve as conditions change. The stake is that a one-time approval process cannot govern a system whose behavior depends on shifting models, data, policies, tools, and user requests. Static sign-off fits static software. Agents require assurance that watches behavior in the field, not just performance in a lab.

Evaluation sets matter because they turn expected behavior into repeatable evidence. They should include normal work, edge cases, adversarial prompts, policy conflicts, and high-impact scenarios. A fraud agent should be tested not only on detection accuracy, but on customer harm, escalation quality, and consistency under changing attack patterns. A hospital agent should be tested not only on prediction, but on whether its recommendations remain clinically and operationally defensible.

Continuous assurance also changes incident response. When an agent goes wrong, the question is not just what failed, but which goal, context, tool, permission, or feedback loop contributed. The decision to make is how much evidence is needed to trust the system at each level of autonomy, and what signals would force a pause, rollback, or redesign.

Static sign-off cannot govern shifting behavior; assurance must watch behavior in the field and loop back.

The capital decision is where to buy speed without buying fragility

The bet is that agents can bend the cost curve of knowledge work and operations. The stake is that early deployment choices can lock in brittle processes, opaque dependencies, and control costs that overwhelm the productivity gain. The unit economics must include not only model and platform costs, but review time, exception handling, audit evidence, incident recovery, customer remediation, and vendor-switching friction.

The S-curve matters. In the early phase, the best returns often come from narrow, high-volume, reversible workflows: claims triage, service-desk routing, sales research, supplier follow-up, maintenance scheduling, document comparison. As confidence and evidence improve, autonomy can expand into more material actions. Moving too slowly leaves competitors with learning advantages. Moving too quickly creates trust debt that is expensive to repay.

The real decisions are portfolio decisions, not technology bets in isolation. Where is the workflow repetitive enough to justify automation? Where is the error cost low enough to learn safely? Where does proprietary data create advantage? Where would a public failure damage trust more than the efficiency gain is worth? Where does keeping optionality matter more than optimizing for a single platform today?

Fund narrow deployments where the business case survives the full cost of controls.
Avoid granting broad tool access before the economic value and failure modes are understood.
Treat escalation capacity as part of the investment, not as spare human bandwidth.
Preserve portability where process knowledge, data, or audit trails would be costly to move later.

The strategic failure is treating agents as another channel

The bet is that agents become a new operating layer: software that can observe, reason, and act across functions. The stake is that legacy governance may underestimate them because they arrive through familiar surfaces such as chat, workflow automation, analytics, or customer service. The interface looks incremental; the operating model is not.

When agents go wrong, the failure usually has a chain: the wrong objective, incomplete context, excessive authority, weak tool controls, poor monitoring, or incentives that reward throughput over judgment. That chain can be managed, but only if the organization treats agent deployment as a change in decision rights. The important map is not a model inventory alone. It is a map of delegated decisions, connected tools, affected stakeholders, and acceptable consequences.

The decision now is where to draw a durable line between experimentation and operation. Experiments can tolerate ambiguity. Operating systems need evidence, limits, accountability, and a way back. The winners will not be those that either ban agents or unleash them everywhere. The advantage will go to institutions that delegate action deliberately, learn faster than their risk accumulates, and keep trust as a constraint rather than a slogan.

Key takeaways

Agents change the risk profile because they act; bad outputs become bad transactions, records, access changes, and customer decisions.
The most important design choice is the autonomy boundary: what the agent may recommend, prepare, execute, or escalate.
Many failures begin with goals and context, not model quality; a system can be accurate against the wrong objective or evidence.
Tool access should be governed by blast radius, reversibility, detectability, and trust impact.
One-time approval is insufficient; agent assurance must be continuous and tied to live behavior.
Capital should flow first to narrow, reversible, high-volume workflows where the full cost of controls still leaves a strong case.

Glossary

Agent: Software that can pursue a goal, choose steps, use tools, and take action with some degree of autonomy.
Autonomy boundary: The line that determines what an agent may do on its own and what requires human review or approval.
Tool invocation: An agent’s use of an external system, API, or workflow to retrieve information or take action.
Prompt injection: A technique that hides malicious or conflicting instructions inside content an agent may read and mistakenly follow.
Blast radius: The scale of harm that can result if an action goes wrong, including money, records, customers, operations, and trust.
Evaluation set: A curated group of scenarios used repeatedly to test whether an agent behaves acceptably under expected and difficult conditions.
Rollback: The ability to reverse, contain, or unwind an agent’s action after an error or unacceptable outcome.