Detecting Regressions Without Ground Truth: Reference-Free Evals and Drift Signals

In a test harness you know the right answer, so a regression is a number that went down. In production you know nothing of the kind: the model answers questions no one has graded, on inputs that drift with the world, and the only thing you can actually observe is the stream of its own outputs. Detecting that the system got worse therefore becomes a two-step inference - first notice that behavior changed, then decide whether the change was bad - and each step has to be built from proxies rather than truth. This article is about doing that inference honestly: which reference-free signals to watch, which statistics turn them into an alarm, how to keep the alarms from crying wolf, and how to keep the proxy tethered to reality.

The Problem: Regression With No Answer Key

Fix the setting precisely. A deployed model produces outputs for a stream of live inputs; there is no reference answer for any of them at the time they are served. You cannot compute accuracy, F1, or exact-match, because all of those need a label. What you can compute is a proxy - a reference-free score, or a summary statistic of the outputs - and you can watch that proxy over time.

This reframes regression detection as change detection. You are not asking is this output correct (unanswerable without truth); you are asking has the distribution of my proxy signal shifted away from a known-good baseline, and if so, in a direction that implies worse quality. The first half of that question is a classical statistical problem with mature tools; the second half is where evals and calibration come in.

Keeping the two halves separate is the discipline the whole article rests on. Conflate them and you will either alarm on harmless change or miss real regressions that a single proxy failed to reflect.

Reference-Free Evals

A reference-free eval scores an output without a gold answer. Three families do most of the work. A rubric judge - typically a language model - scores an output against explicit criteria (grounded, on-task, safe); it is flexible and runs online, and, as Zheng and colleagues document, it approximates human preference well while carrying biases toward verbosity, position, and self-agreement that must be calibrated out. Self-consistency samples the same input several times and measures agreement; low agreement is a signal of instability even when no answer is known to be right. Property checks are cheap deterministic predicates - did the output cite a source, did it refuse a disallowed request, is it valid JSON - that need no model at all.

None of these measures truth; each measures a correlate of quality, and the HELM stance of Liang and colleagues applies directly: prefer a vector of complementary reference-free metrics over a single blurred score, because any one correlate can be gamed or can miss a failure mode the others catch.

The practical move is to run the cheap property checks on everything, the rubric judge on a sample, and self-consistency where instability is the risk - a layering the final section formalizes.

Distributional Signals You Get for Free

Below the evals sit signals so cheap you should always collect them, because they are byproducts of serving. Refusal rate shifts when a safety layer or the model's willingness changes. The distribution of output length reveals truncation or a drift toward verbosity. An embedding summary of outputs - a centroid and a spread - moves when the topic or style of what the model produces changes. The mix of tool calls shifts when an agent's behavior changes. None of these judges quality directly, but all of them move when behavior moves, and they cost essentially nothing.

Their value is coverage and speed: they are computed on one hundred percent of traffic with no extra model call, so they catch gross shifts immediately and for free, long before an expensive eval sample would accumulate enough data to notice. Their weakness is direction - a change in output length is a signal that something moved, not that it got worse.

That division of labor - cheap signals for detection, evals for direction - is why a real system watches both, and why the table below pairs each signal with its cost and what it actually detects.

Change Detection, Formally

Turning a proxy stream into an alarm is a solved statistical problem with three workhorses. A two-sample test compares a reference window (known-good) against a current window and asks whether they came from the same distribution. The Kolmogorov-Smirnov statistic takes the largest gap between the two empirical cumulative distributions - good for a continuous score like eval quality or output length. The Population Stability Index compares binned distributions and is the standard drift index for categorical or bucketed signals like tool-call mix. For a running signal where you want to catch a shift as fast as possible, the cumulative-sum (CUSUM) procedure of Basseville and Nikiforov accumulates evidence of a downward shift and alarms when it crosses a threshold.

Each has a natural fit: CUSUM for fast online detection of a drop, KS for a continuous two-window comparison, PSI for bucketed distributions. What they share is the shape of the answer - a distance between a baseline and now, compared against a threshold - and the same fundamental knob, the threshold, which sets how large a shift must be before it counts.

The threshold is not a free parameter; it is a decision about how much you are willing to be wrong, which the next section makes explicit.

The Base-Rate Problem and a False-Alarm Budget

Every detector trades two errors: a lower threshold catches real regressions sooner but fires more false alarms, a higher one is quieter but slower. For a sequential detector this is captured by the average run length - the expected number of observations between false alarms when nothing is wrong - and the false-alarm rate is its reciprocal. Set the threshold by choosing an acceptable time between false alarms, not by intuition.

The problem compounds because you are watching many signals at once. If each of twenty proxies independently false-alarms even rarely, the system as a whole cries wolf constantly - the classic multiple-comparisons trap. The fix is to treat false alarms as a budget: decide the tolerable rate of spurious pages, then allocate it across signals, require corroboration (two independent signals, or a cheap signal confirmed by an eval) before escalating, and aggregate related alarms into one.

This is the measurement rigor that separates a drift monitor from a noise generator. An alarm that fires once a day and is wrong four times out of five will be muted by its operators within a week, which is worse than no monitor at all.

Calibrating the Proxy Against Truth

A drift detector on a proxy tells you the proxy moved. Whether that corresponds to a real quality change is a separate question you can only answer by occasionally touching ground truth. The discipline is to draw a small labeled sample periodically - human ratings, or a stronger reference model treated as an anchor - and check that the proxy tracks it: does a drop in the rubric-judge score actually coincide with worse human-rated quality? If it does, the proxy is calibrated and you can trust its alarms; if it does not, you are detecting drift in the judge, not in the product.

This closes a loop the eval literature is explicit about. Zheng and colleagues calibrate model-based judges against human labels precisely because the judge is an instrument with drift and bias of its own; the HELM methodology's insistence on multiple metrics is a hedge against any single proxy decoupling from reality. Calibration is not a one-time step - the judge model, the input mix, and the product all move, so the calibration sample is itself a recurring measurement.

The payoff is that alarms mean something. A calibrated proxy that drops is evidence of a real regression; an uncalibrated one is a rumor.

A Drift-Signal Ladder

The signals differ by orders of magnitude in cost, so the right architecture is a ladder that escalates only when a cheaper rung fires. At the base sit the always-on distributional signals - refusal rate, length, embeddings, tool mix - computed on all traffic for free. When one of them trips, the middle rung runs reference-free evals on a sample of the affected slice to add direction. Only when the evals confirm a likely regression does the top rung spend the expensive resource: human spot-checks on the specific flagged cases, which also feed the calibration set.

The ladder makes the economics work. You get the coverage of watching everything (the cheap base is always on) and the precision of human judgment (the expensive apex is rarely invoked), without paying for a judge on every response or a human on every judge. It also gives every alarm a natural triage path: base signal detects, evals localize and direct, humans confirm.

This structure is the article's synthesizing artifact - not a specific tool, but an ordering principle: detect cheaply and broadly, confirm selectively, and reserve truth for the few cases that earn it.

Threats to Validity, and the Honest Limit

The sharpest trap is confusing covariate shift with regression. If your input distribution changes - a new customer segment, a seasonal topic - your proxies will move even though the model is unchanged and just as good. Distinguishing the two requires asking both questions: did the distribution change, and did quality drop on comparable inputs. A drift alarm with no quality drop on matched inputs is covariate shift to adapt your baseline to, not a regression to roll back.

The other limits follow from the same root. A detector finds change, not badness - direction always comes from an eval, never from the distance statistic alone. Proxies are subject to Goodhart's law and to calibration drift, so any signal you gate on must be rotated and re-calibrated. And seasonality and multiplicity manufacture false alarms unless budgeted. Stated plainly: this machinery reliably tells you that something moved and, with a calibrated eval, whether it moved for the worse - which is exactly as much as is knowable without an answer key, and enough to catch real regressions before users do.

Build it as a ladder, calibrate the proxies, budget the false alarms, and always pair a change signal with an eval for direction. That is regression detection without ground truth, done honestly.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

function drift_check(ref_bins, cur_bins, quality_stream):
    # distributional drift on a bucketed signal
    psi = 0
    for i in buckets:
        p_ref = ref_bins[i]; p_cur = cur_bins[i]
        if p_ref > 0 and p_cur > 0:
            psi += (p_cur - p_ref) * ln(p_cur / p_ref)
    dist_alarm = psi > PSI_THRESHOLD
    # fast online guard on a quality score (CUSUM for a drop)
    S = 0
    for l in quality_stream:            # l = reference-free eval score
        S = max(0, S + (K_REF - l))
        if S > H: quality_alarm = true
    return corroborate(dist_alarm, quality_alarm)   # page only if both / calibrated

For a sampled input, request K independent answers at nonzero temperature.
Reduce each answer to its key claim(s). Compute agreement = fraction of the K that
share the majority claim. Emit agreement in [0,1] as the signal; low agreement means
the model is unstable on this input even though no answer is known to be correct.
Alarm on a downward shift in the agreement distribution, then route low-agreement
cases to human review. Do NOT treat the majority answer as ground truth.

Glossary

Reference-free eval

A score of an output computed without a gold answer, via a rubric judge, self-consistency, or deterministic property checks.

Self-consistency

Sampling the same input several times and measuring agreement; low agreement signals instability without needing a correct answer.

CUSUM

A cumulative-sum change detector that accumulates evidence of a shift and alarms when the accumulation crosses a threshold.

Kolmogorov-Smirnov statistic

The largest gap between two empirical cumulative distributions, used to test whether two samples share a distribution.

Population Stability Index (PSI)

A binned distance between a reference and a current distribution, the standard drift index for categorical or bucketed signals.

Average run length (ARL)

The expected number of observations between alarms; ARL under no change sets the false-alarm rate as its reciprocal.

Covariate shift

A change in the input distribution that moves proxy signals even though model quality on comparable inputs is unchanged.

References

1. Basseville & Nikiforov, Detection of Abrupt Changes: Theory and Application (1993)
2. Liang et al., Holistic Evaluation of Language Models (HELM)
3. Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
4. NIST AI 100-1, Artificial Intelligence Risk Management Framework (AI RMF 1.0)
5. NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative AI Profile
6. NIST SP 800-137, Information Security Continuous Monitoring (ISCM)