Excessive Agency — AI Threat Basics

A tool waits; an agent acts. That small difference changes the shape of risk. A calculator gives an answer and stops. A software agent can read a message, choose a next step, call a tool, change a record, send a reply, and try again if it fails. Agency is useful because it turns intention into action. Excessive agency is dangerous because it lets action outrun judgment, oversight, and need.

What agency means

Agency is the capacity to pursue a goal by choosing and taking actions. A thermostat has a little agency: it senses temperature, compares it with a target, and turns heating or cooling on or off. A human employee has much more agency: they interpret ambiguous situations, use many tools, negotiate priorities, and decide when to ask for help.

In software, an agent is a system that can observe some part of the world, decide what to do next, and act through tools. The tools may be harmless, such as searching a document, or consequential, such as sending money, deleting data, changing a customer record, or emailing a client.

Agency is not the same as intelligence. A system can be clever but passive, like a model that drafts text for a person to review. It can also be simple but agentic, like a script that automatically disables accounts after a rule is triggered. The risk comes from the combination of autonomy, tools, permissions, and real-world consequences.

When agency becomes excessive

Agency becomes excessive when a system has more freedom than the task needs or more power than its supervisors can safely govern. The word excessive matters. Agency is not bad by itself. A delivery robot needs enough autonomy to cross a hallway. A fraud-monitoring system needs enough permission to pause a suspicious transaction. The problem is granting broad action rights for narrow jobs.

There are four common forms. The system may have too much scope, meaning it can operate across too many users, records, systems, or business processes. It may have too much speed, meaning it can make many changes before anyone notices. It may have too much discretion, meaning its goal is vague and it fills in the blanks on its own. Or it may have too much irreversibility, meaning its actions are hard to undo.

A useful test is simple: if the system misunderstands the situation, what can it do before a person or another control stops it? The larger that answer is, the more agency you have given it.

⚠️

The core risk. Excessive agency lets a mistaken or manipulated system convert a bad interpretation into a real action.

Agency becomes excessive through too much scope, speed, discretion, or irreversibility.

Why modern artificial intelligence changes the discussion

Artificial intelligence is a broad term for software that performs tasks we associate with human judgment, such as recognizing patterns, classifying information, or generating language. A large language model (LLM) is a kind of artificial intelligence trained to predict and produce text. On its own, an LLM that only drafts a paragraph has limited agency. It produces output, but it does not necessarily act.

The risk changes when an LLM is connected to tools. A tool can be an application programming interface (API), which is a defined way for software systems to request actions from each other. Through APIs, an agent might look up invoices, open support tickets, update a calendar, approve a refund, or publish a document. The LLM supplies flexible reasoning and language; the tools supply leverage.

This is why the same model can be low risk in one setting and high risk in another. A model that summarizes a policy manual for a lawyer is assisting judgment. The same model with permission to interpret policy, approve exceptions, and notify customers is exercising judgment.

How excessive agency shows up in real systems

Excessive agency often looks reasonable at first because each added permission is convenient. A customer support assistant needs to read tickets, so it gets access to the ticketing system. It needs context, so it gets customer records. It needs to help, so it can issue refunds. It needs to finish the conversation, so it can send emails. At some point, a helper has become an operator.

The failure may not be dramatic. It may be a quiet sequence of plausible steps. The agent misreads sarcasm as approval. It treats a draft as final. It updates the wrong account because two names are similar. It follows an instruction hidden inside an email. It retries an action that was already completed. Each step looks locally defensible; the combined effect is wrong.

This is different from ordinary software bugs. Traditional software usually fails within paths its designers wrote. Agentic systems can compose actions in ways their designers did not explicitly enumerate. That flexibility is the value, but it also makes the boundary of expected behavior harder to draw.

A scheduling agent that can invite anyone inside and outside the organization, exposing confidential meeting titles.
A finance agent that can both create a vendor and approve payment, removing separation of duties.
A coding agent that can run commands, read secrets, and push changes directly to production.
A sales agent that can discount prices and send binding offers without review.

Each convenient permission adds up until a read-only helper quietly becomes an operator.

The control problem is not only technical

A common mistake is to treat excessive agency as a model-quality problem: if the model becomes more accurate, the risk will go away. Better models help, but they do not remove the need for boundaries. Even a capable employee does not receive every permission in the company. Competence and control are separate questions.

The practical control problem has three layers. First, define what the system is allowed to decide. Second, define what it is allowed to do without approval. Third, define how you will notice, stop, and recover from a bad action. These are governance questions as much as engineering questions.

A human in the loop is a person who must review or approve an action before it happens. This can be effective, but only if the person has enough context, time, and authority to disagree. A tired reviewer clicking approve on a hundred opaque recommendations is not meaningful oversight.

⚠️

Review is not magic. Human approval only reduces risk when the reviewer can understand the action and is expected to stop it when necessary.

Control has three layers: what it may decide, what it may do unapproved, and how you recover.

Decisions leaders and practitioners must make

The central leadership question is not, Can this system do the work? It is, How much unsupervised consequence are we willing to allow? That question belongs in product design, security review, legal review, operations planning, and incident response.

Practitioners need to translate that question into concrete settings. What data can the agent read? What tools can it call? Which actions are read-only, which are reversible, and which require approval? How many actions can it take per minute, per user, or per case? What evidence is logged? Who is alerted when behavior looks unusual? Who can shut it off?

A useful pattern is to start with the narrowest possible agency and expand only where evidence justifies it. This is the principle of least privilege: a system should have only the permissions it needs to do its current job. In agentic systems, least privilege should apply not only to data access, but also to goals, tools, rate limits, and autonomy.

Designing for appropriate agency

Appropriate agency is agency matched to purpose. The design task is to give the system enough freedom to be useful and enough constraint to be safe. That balance will vary by domain. A travel agent suggesting flights can tolerate some wrong guesses. A medical, financial, or infrastructure agent needs stricter limits because the blast radius is larger.

Blast radius means the amount of harm a failure can cause before it is contained. Reducing blast radius is often more reliable than trying to prevent every mistake. You can limit the number of records touched, cap the size of transactions, require approval for unusual cases, separate duties across systems, test in a sandbox, and make actions reversible where possible.

Good designs also make agency visible. Logs should show what the agent saw, what it decided, which tool it used, and why the action was permitted. Users should be able to tell whether they are receiving a suggestion, approving a prepared action, or delegating authority. Ambiguity about who is acting is itself a risk.

Prefer read-only access until write access is clearly necessary.
Separate recommendation from execution for high-impact actions.
Use approval gates for irreversible, unusual, or high-value changes.
Apply rate limits so a loop cannot cause large damage quickly.
Record enough context to investigate and reverse bad actions.

✅

Right-sized agency. The safest useful agent is not the smartest one; it is the one whose authority fits the work.

Key takeaways

Agency is the ability to pursue a goal by choosing and taking actions.
Excessive agency occurs when autonomy, scope, speed, or irreversibility exceeds the task’s real need.
Artificial intelligence becomes more consequential when connected to tools, permissions, and repeated action loops.
Human review helps only when reviewers have context, time, and authority to stop bad actions.
Leaders should decide how much unsupervised consequence a system may create before deploying it.
Least privilege, approval gates, rate limits, logging, and reversibility keep agency matched to purpose.

Appropriate agency matches the blast radius: suggest for low stakes, autonomy only where justified.

Glossary

Agency: The capacity to pursue a goal by choosing and taking actions.
Agent: A software system that can observe a situation, decide what to do next, and act through tools.
Autonomy: The degree to which a system can operate without a person making each decision.
Tool: A function, service, or interface an agent can use to retrieve information or take action.
Application programming interface: A defined way for one software system to request data or actions from another software system.
Human in the loop: A control pattern in which a person must review or approve an action before it happens.
Least privilege: The principle that a person or system should have only the permissions needed for the current job.
Blast radius: The amount of harm a failure can cause before it is contained.