An agent system is not evaluated by asking whether it sounds clever. It is evaluated by asking whether it can do the right work, under real constraints, often enough to be trusted. That sounds simple, but agents make evaluation harder than ordinary software because they choose steps, call tools, handle messy language, and sometimes take different paths to the same answer. Good evaluation turns that uncertainty into evidence.

Start with the job, not the model

An agent system is software that uses a model, usually a large language model (LLM), to decide what to do next. It may read instructions, call a tool, search a knowledge base, write a message, update a record, ask a person for help, or stop. The model is important, but the system is the thing you must evaluate.

First principles help. A system is useful only if it changes an input into a desired outcome. For an agent, the input might be a customer request, a document, an alert, or a business question. The desired outcome might be a resolved ticket, a correct summary, a safe recommendation, or a completed workflow.

Before you measure anything, write down the job in plain language. What task should the agent complete? Who depends on the result? What counts as success? What must never happen? If those answers are vague, the evaluation will be vague too.

Define success as observable behavior

A useful evaluation does not depend on a feeling. It names behaviors that can be observed. Did the agent answer the question? Did it use the right source? Did it avoid exposing personally identifiable information (PII)? Did it ask for approval before making a change? Did it finish within the time and cost limits?

Some criteria are binary: the order was either canceled or it was not. Others are graded: a summary may be complete, partially complete, or misleading. Graded measures are normal in agent evaluation because language tasks often have more than one acceptable answer.

The best measures are tied to consequences. A harmless typo in an internal draft may be a minor issue. The same typo in a medication instruction could be severe. Severity matters because evaluation should guide decisions, not just produce a score.

  • Task success: whether the requested job was completed.
  • Factuality: whether claims are supported by trusted evidence.
  • Safety: whether the agent avoided harmful, forbidden, or policy-breaking behavior.
  • Reliability: whether performance remains stable across repeated and varied cases.
  • Latency: how long the user or downstream system waits.
  • Cost: how much compute, model usage, and tool usage the task consumes.
📌
Measure outcomes. A polished answer that does not complete the task is a failure, even if it reads well.

Build test cases from real work

A test case is a small, repeatable example used to learn how the system behaves. For an agent, a test case should include the input, the expected outcome, the relevant policy or source material, and any constraints such as time, permissions, or required approval.

Start with real examples if you can. Production logs, support tickets, call transcripts, forms, emails, and analyst questions contain the variety that synthetic examples often miss. Remove or protect sensitive data before using them. If real examples are unavailable, create scenarios with the people who know the work best.

A good test set is not only a list of easy happy paths. It includes routine cases, edge cases, ambiguous requests, missing information, malicious prompts, tool failures, and examples where the right answer is to refuse, escalate, or ask a clarifying question.

Look inside the run

Traditional software testing often checks the final output. Agent evaluation needs more visibility because the path matters. An agent may reach a correct answer using an unsafe source, make an unauthorized tool call, ignore an instruction, or retry until it accidentally succeeds. You need to inspect the run, not just the result.

A trace is a record of what happened during one run. It can show the input, retrieved documents, model messages, tool calls, tool results, decisions, errors, timing, and final output. Traces make failures explainable. Without them, teams argue about symptoms instead of causes.

Tracing is also how you separate model problems from system problems. A wrong answer might come from a weak prompt, stale data, missing permissions, a broken application programming interface (API), an overly broad tool, or an inadequate model. The fix depends on the source of the failure.

⚠️
Keep evidence. If you cannot replay or inspect important runs, you cannot confidently improve the agent.
A trace records the whole path of one run: input, retrieval, tool calls, and final output. Inside The Run Input Request in Retrieval Sources read Tool calls Actions taken Output Final answer
A trace records the whole path of one run: input, retrieval, tool calls, and final output.

Use several kinds of judgment

No single judge is enough. Exact matching works when there is one correct value, such as an invoice number. Programmatic checks work when a rule can be encoded, such as whether the answer includes a required citation. Human review works when quality depends on context, tone, or professional judgment.

Model-based judging can help at scale, where another model reviews the agent output against a rubric. This is useful, but it is not magic. A model judge can be biased, inconsistent, or too forgiving of fluent errors. Treat it as one instrument, calibrate it against human reviewers, and monitor drift over time.

The strongest evaluations combine methods. Use automated checks for what machines can verify, human review for high-stakes judgment, model judges for scale, and production monitoring for what only appears in live use.

  • Exact checks for fields, calculations, formats, and required steps.
  • Reference checks against ground truth, trusted documents, or approved answers.
  • Rubric-based human review for usefulness, completeness, and risk.
  • Model-based judging for broad coverage, with human calibration.
  • Live monitoring for failures that test sets did not predict.
Strong evaluation combines exact matching, programmatic checks, human review, and model judges. Kinds Of Judgment Automated Human Rigid Judgment Exact match One right value Programmatic Encoded rule Human review Context & tone Model judge Scale with rubric
Strong evaluation combines exact matching, programmatic checks, human review, and model judges.

Evaluate the boundaries, not only the average

Average performance can hide danger. An agent that succeeds in 90 percent of cases may still be unacceptable if the remaining 10 percent includes privacy leaks, financial errors, or unsafe advice. Evaluation must focus on boundaries: the places where the system should slow down, refuse, ask for help, or stop.

Guardrails are controls that constrain behavior. They may check inputs, limit tools, require permissions, filter outputs, enforce policy, or route uncertain cases to a person. Security controls also matter. For example, identity and access decisions may depend on Microsoft Entra ID or another identity provider, and the evaluation should confirm that the agent respects those permissions.

Boundary testing should include adversarial prompts, conflicting instructions, missing data, excessive confidence, and tool misuse. The goal is not to prove the agent can never fail. The goal is to know how it fails, whether the failure is contained, and how quickly the organization can detect it.

⚠️
Averages mislead. For high-impact tasks, the worst credible failure may matter more than the mean score.
Boundary testing checks that uncertain cases slow down: proceed within limits, otherwise refuse or escalate. At The Boundary Uncertain case Within limits? Proceed Act normally Refuse or escalate Route to a person Yes No
Boundary testing checks that uncertain cases slow down: proceed within limits, otherwise refuse or escalate.

Turn evaluation into an operating loop

Evaluation is not a gate you pass once. Agent systems change when prompts change, tools change, data changes, policies change, models are updated, or users discover new ways to ask for work. A good team treats evaluation as an operating loop.

The loop begins with observations from tests and production. The team diagnoses failures, changes the system, reruns the evaluation, compares results with earlier baselines, and then decides whether to release. Platforms such as Microsoft Foundry can support parts of this lifecycle, but the discipline is independent of any tool: define the evidence, collect it consistently, and use it to make decisions.

Regression testing is essential. A change that improves one task can quietly damage another. Keep a stable set of important cases and rerun them whenever you change prompts, tools, policies, retrieval logic, or model versions.

Evaluation is a loop: observe, diagnose, change, and re-evaluate against a stable baseline. Operating Loop Observe Diagnose Change Re-evaluate baseline
Evaluation is a loop: observe, diagnose, change, and re-evaluate against a stable baseline.

Make the decision explicit

Leaders and practitioners need evaluation because they must make real choices: whether to launch, limit scope, add human review, change a model, buy more latency with better accuracy, reduce cost, expand tool access, or stop the project. Evaluation should make those trade-offs visible.

A practical release decision combines evidence and risk appetite. A low-risk drafting assistant may be acceptable with modest accuracy if users review every output. An autonomous refund agent needs stricter task success, stronger audit trails, tighter permissions, and clear escalation. The same score can mean different things in different business contexts.

The best evaluation reports are short and decision-oriented. They show the tasks tested, the pass criteria, the main failure modes, the trend since the last version, the remaining risks, and the recommendation. If the evidence does not support a decision, the honest answer is to run a better evaluation before expanding use.

  • Launch broadly when success is high, failures are low impact, and monitoring is ready.
  • Launch narrowly when value is clear but some cases still need containment.
  • Require human approval when the agent can affect money, rights, safety, or customer trust.
  • Hold release when failures are severe, unexplained, or hard to detect.
  • Retire or redesign when the task is poorly suited to an agent or the economics do not work.
✅
Evidence enables judgment. Evaluation does not remove leadership judgment; it gives that judgment a factual base.

Key takeaways

  • Evaluate the whole agent system, including tools, data, permissions, traces, and final outcomes.
  • Define success as observable behavior tied to real consequences.
  • Use realistic test cases that include routine work, edge cases, abuse, failures, and escalation.
  • Inspect traces so you can explain failures and choose the right fix.
  • Combine exact checks, human review, model-based judging, and live monitoring.
  • Use evaluation to make explicit release, scope, safety, cost, and ownership decisions.

Glossary

Agent system
Software that uses a model to choose steps, call tools, and work toward a goal.
Large language model (LLM)
A model trained on large amounts of text that can generate and interpret language.
Tool
A function, service, database, or application programming interface that an agent can call to take action or get information.
Application programming interface (API)
A defined way for software systems to request data or actions from each other.
Trace
A record of the steps, messages, tool calls, data, timing, and results from one run of an agent system.
Ground truth
A trusted reference answer or source used to judge whether an output is correct.
Guardrail
A control that limits or redirects system behavior to reduce risk.
Regression testing
Rerunning important tests after a change to check that previously working behavior still works.