Building an Eval Harness — Production LLM Eval & Observability

A model demo answers one question: can this system work once, in front of us, right now? An eval harness answers a harder and more useful question: does this system keep working across the cases we care about, after the prompt changes, after the model changes, after the data changes, and after real users find the edges? That shift is the difference between admiring an artificial intelligence system and operating one.

What an eval harness is

An evaluation harness, usually shortened to eval harness, is a small system that repeatedly tests an artificial intelligence, or AI, application against known cases and records the results. The application might be a chatbot, a document summarizer, a search assistant, an agent that calls tools, or any other system whose behavior is partly shaped by a model.

The word harness matters. A harness holds something in place so it can do useful work safely. An eval harness does the same for a model-based system. It controls the inputs, runs the system in a consistent way, measures the outputs, and makes the results visible enough that people can make decisions.

Without a harness, teams tend to rely on anecdotes. Someone tries five examples, decides the new prompt feels better, and ships it. That may be reasonable for exploration, but it is not enough for production. A harness gives the team a shared, repeatable way to ask whether a change improved the product, harmed it, or simply moved the errors around.

Start with the question, not the metric

The first mistake in evaluation is to begin with a clever metric. The better starting point is a plain question. For example: does the assistant answer from approved sources? Does it refuse unsafe requests? Does it cite the right document? Does it finish within three seconds? Does it preserve the meaning of the source text?

A metric is only useful if it is tied to a decision. If a score falls from 92 to 88, what happens? Does the team block the release, investigate one slice of traffic, or accept the trade-off because latency improved? If nobody knows, the metric is decoration.

For a first harness, choose a small number of questions that map to real product risk. Most teams need some mix of correctness, safety, usefulness, cost, and speed. The exact mix depends on the job the system is being hired to do.

Correctness: the answer is factually right for the task.
Grounding: the answer is supported by the provided sources.
Safety: the system avoids prohibited or harmful behavior.
Completeness: the answer covers the required parts of the request.
Latency: the system responds within an acceptable time.
Cost: the system uses an acceptable amount of model and tool capacity.

📌

Make metrics actionable. A metric should tell the team what decision is now easier to make.

Build the test set like a product asset

The test set is the heart of the harness. It is a collection of test cases: inputs, context, expected behavior, and any labels needed for scoring. In classical software, a test might assert that a function returns one exact value. In AI systems, the expected behavior is often broader: the response may be phrased differently but still be acceptable.

A good test set has ordinary cases, hard cases, and cases that once failed. Ordinary cases protect the core experience. Hard cases explore the boundaries. Past failures prevent regression, which is when a system loses behavior that used to work.

Do not wait for a perfect golden dataset. A golden dataset is a carefully reviewed set of examples treated as a trusted reference. It is valuable, but it grows over time. Start with 30 to 100 meaningful examples, label them carefully, and add cases whenever real usage teaches you something.

⚠️

Small is fine. A small, well-labeled set usually teaches more than a large set nobody trusts.

Ordinary, hard, and previously failed cases together protect, stretch, and guard the system.

Separate the runner from the scorers

An eval harness has two jobs that should be kept separate. The runner executes the system under test. It sends the input, supplies the same context the product would use, captures the answer, records timing, and stores any tool calls or retrieved documents. The scorers examine what happened and assign results.

This separation keeps the harness flexible. You may run the same outputs through several scorers: an exact-match check, a rule that verifies citations, a human review queue, and a judge model. A judge model is another model asked to evaluate an output against instructions or a rubric. It can be useful, but it should be treated as an imperfect measurement instrument, not as truth.

In Microsoft Foundry, for example, a team might run an application against a set of prompts, call tools from Foundry Tools, authenticate access through Microsoft Entra ID, and store traces for later review. The principle is the same in any stack: keep execution, scoring, and reporting loosely coupled.

The runner executes the system and records output; separate scorers then assign results.

Choose scoring methods by failure type

No single scoring method fits every failure. Exact checks are best when the answer must contain a known value, such as a code, date, identifier, or classification. Rule-based checks are best when the output must follow a format, include a citation, avoid a phrase, or stay under a length. Human review is best when judgment, policy, or domain expertise matters.

Judge models are helpful for fuzzy questions, such as whether a summary preserved the important facts or whether an answer followed a multi-part instruction. They are also risky because they can be inconsistent, biased by wording, or blind to domain details. The safest pattern is to use a clear rubric, keep examples of good and bad judgments, and periodically compare judge scores with human review.

For many systems, the strongest evaluation is a layered one. A response may need to pass a schema check, a citation check, a safety check, and a usefulness score. This is closer to how users experience quality: a fast answer that is unsafe is not good, and a correct answer in the wrong format may still break the workflow.

Use exact checks for values that should not vary.
Use rules for structure, format, policy flags, and simple constraints.
Use retrieval checks when the system must use approved sources.
Use judge models for graded language qualities that are hard to express as rules.
Use human review to calibrate the rest of the harness and settle high-risk questions.

⚠️

Do not hide uncertainty. A judge model score should not be presented as objective truth unless it has been calibrated against trusted review.

A known exact answer allows automated checks; fuzzy judgment needs human or judge review.

Make results comparable over time

An eval harness becomes valuable when it produces comparable results. That means the team can look at a run from last week and a run from today and understand what changed. To make that possible, record the model version, prompt version, code version, data source version, tool configuration, timestamp, and scoring version for every run.

The harness should store both summary scores and raw evidence. A dashboard that says correctness is 84 percent is useful, but it is not enough. Engineers need to inspect the failed cases, the model output, the retrieved context, tool calls, scorer notes, and any human comments. Summaries guide attention; traces explain causes.

Teams often wire the harness into continuous integration and continuous delivery, or CI/CD, which is the practice of automatically testing and preparing changes for release. Not every eval must block a deployment. Some checks should be release gates; others should run nightly to watch for drift, which is a change in behavior caused by shifting inputs, data, models, or usage.

Use the harness to make real decisions

The point of an eval harness is not to produce a beautiful report. It is to make choices less subjective. A product leader may need to choose between a cheaper model and a more capable one. An architect may need to decide whether to add retrieval, change a prompt, introduce a tool, or split one broad assistant into narrower skills. A risk owner may need to decide which failures are acceptable and which must block launch.

The harness does not make those decisions alone. It supplies evidence. For example, a smaller model may reduce cost by 40 percent while lowering grounded answers by 3 percent on ordinary cases and 18 percent on legal questions. That result does not say yes or no by itself, but it makes the trade-off visible. The team can then decide whether to route legal questions differently, improve retrieval, or keep the larger model.

Good evaluation also protects learning speed. Teams that lack a harness become cautious because every change feels uncertain. Teams with a harness can move in smaller, safer steps: change one thing, run the tests, inspect the failures, and decide.

A practical first version

A first eval harness should be boring on purpose. Put the test cases in a simple, reviewable format. Write a runner that calls the same application interface the product uses. Store every input, output, trace, and score. Produce a report that shows totals, changes from the previous run, and the worst failures. Keep the design simple enough that the whole team understands it.

Start with a baseline run before you improve anything. The baseline is the reference point that makes later movement visible. Then add one change at a time: a prompt edit, a retrieval setting, a model version, or a tool change. If several things change at once, the harness may tell you that quality moved, but not why.

The harness will never be finished. New user behavior, new documents, new policies, and new model capabilities will all create new cases to test. That is not a flaw. It is the maintenance work that turns evaluation from a one-time launch task into an operating discipline.

Define the decisions the harness should support.
Collect a small set of representative and risky cases.
Label expected behavior clearly enough for another person to review.
Build a runner that captures outputs, traces, timing, and versions.
Add simple scorers before adding complex ones.
Compare every change against a baseline and inspect the failures.

✅

Aim for repeatability. The first win is not perfect scoring; it is the ability to run the same evaluation again and trust the comparison.

Key takeaways

An eval harness is a repeatable system for testing an AI application, not a one-time review.
Start with decisions and risks, then choose metrics that make those decisions easier.
The test set is a product asset and should grow from real cases, hard cases, and past failures.
Separate running the system from scoring the outputs so each part can improve independently.
Use different scoring methods for different failure types, and calibrate judge models with human review.
Store versions, traces, and raw evidence so results remain comparable over time.

A simple harness runs test cases, stores evidence, and reports totals and changes.

Glossary

Eval harness: A repeatable system that runs an AI application on known cases, scores the results, and records evidence for decisions.
Test case: A single example used in evaluation, usually including an input, context, expected behavior, and labels for scoring.
Golden dataset: A carefully reviewed set of examples treated as a trusted reference for evaluating system behavior.
Metric: A measurement used to summarize some aspect of system behavior, such as correctness, latency, safety, or cost.
Judge model: A model used to grade or compare another model's output according to a rubric.
Regression: A failure where a system loses behavior that previously worked.
Drift: A change in system behavior over time caused by shifting inputs, data, models, configuration, or user patterns.
Ground truth: The trusted answer or label used as the reference when judging whether a system output is correct.
Continuous integration and continuous delivery: An engineering practice where changes are automatically tested and prepared for release through a repeatable pipeline.