The Capability-Measurement Paradox — Frontier AI Security: Boundary II

The hardest part of measuring a frontier artificial intelligence (AI) system is not writing the benchmark. It is deciding how to test a capability that is dangerous precisely because it works. If the model is kept behind its normal safety systems, the test may show only that the safety systems intervened. If those systems are relaxed, the test becomes more truthful and more hazardous at the same time. That is the capability-measurement paradox: the better you measure the danger, the more carefully you must contain the measurement itself.

The paradox in plain language

A safety evaluation asks a simple question: what could this model do if it tried? For ordinary software, that question is difficult but familiar. For a frontier AI model, it becomes strange, because the model may be surrounded by safeguards designed to prevent exactly the behavior the evaluator is trying to observe.

If those safeguards stay on, the result can be reassuring but incomplete. The evaluator learns what happened under restraint. That is not the same as learning the model’s maximum capability. If the safeguards are relaxed, the evaluator can get a clearer measurement, but only by creating a test condition in which the model is less safe than it would normally be.

The paradox is not a paperwork problem or a failure of imagination. It follows from the meaning of maximum capability. To measure the top of a dangerous range, the test must allow the system to approach that range.

⚠️

The central tension. A test can be safer because it is restrained, or more revealing because it is unrestrained, but it cannot be both in the same simple way.

Restrained tests are safe but incomplete; measuring maximum capability means relaxing the very safeguards under study.

Why restrained tests can mislead

A safety system is supposed to block harmful requests, refuse dangerous instructions, limit tools, filter outputs, or interrupt risky action sequences. If that system works during an evaluation, the model may appear less capable than it really is. The test has measured the combined behavior of the model and its guardrails, not the model’s underlying ability.

This distinction matters because leaders often need to know the ceiling, not only the normal operating behavior. A model that refuses to write exploit code in production may still understand how to do it. A model that cannot reach the internet during ordinary use may still be able to plan a multi-step cyber operation if tool access is later granted. A model that stops after one refused instruction may behave differently when placed inside a long automated task with memory and retries.

A restrained test is not useless. It can show whether the deployed product blocks known classes of misuse. But it is the wrong instrument for answering whether the model has the capability in the first place.

A product-safety test asks whether the deployed system refuses or contains a harmful request.
A capability test asks whether the model could perform the harmful task if the restraints were absent or bypassed.
Confusing those two questions creates false confidence, because a good refusal can hide a strong underlying capability.

Why unrestrained tests are genuinely different

Unrestrained does not mean careless. It means the evaluation deliberately removes or relaxes some controls so the model can attempt the task. The evaluator might disable refusal behavior, provide more complete tool access, allow longer planning horizons, or run the model as an agent that can take actions over time.

At that point the evaluation is no longer just observing a safe product. It is temporarily instantiating a less safe configuration of the system. That configuration may be necessary for honest measurement, but it changes the risk profile of the test. The model may now produce instructions that should not leave the lab, operate tools in unexpected ways, search for ways around artificial limits, or combine small permitted steps into a larger hazardous chain.

This is why serious evaluations are not merely benchmark jobs. They are controlled experiments with security consequences. The dangerous capability may not be only the final answer the model prints. It may be the path it discovers, the tools it invokes, the data it touches, and the persistence it shows when blocked.

⚠️

Do not confuse access with intent. A model does not need malicious intent for an unrestrained test to be risky; capability, tools, and a goal can be enough.

The sandbox is the usual answer

The standard way to handle the paradox is containment. Evaluators run the unsafe configuration inside a sandbox: an isolated environment built so the model can attempt the task without reaching real systems, real users, real secrets, or the public internet. The model may be dangerous inside the box, but the box is supposed to make the experiment safe overall.

This is sound in principle. Sandboxes are common in malware analysis, exploit testing, and other security work where the object under study cannot be trusted. The same discipline applies to frontier AI evaluations: fake targets instead of production systems, synthetic credentials instead of real secrets, disposable machines instead of persistent infrastructure, and strict network rules instead of open access.

But the sandbox is not decorative. It is the load-bearing control. If it holds, the evaluator learns from an unsafe model without letting the unsafe behavior matter outside the lab. If it fails, the evaluation has created the very condition it was meant to study and allowed it to escape.

A sandbox swaps real assets for safe stand-ins so an unsafe configuration can be studied without real harm.

The circular trap

Containment has to be sized for the thing it is meant to contain. That is easy when the thing is well understood. It is much harder when the evaluation exists because the capability is not yet known. To design the sandbox, the team must make assumptions about what the model can do. To validate those assumptions, the team must run the test.

This creates the circular trap. The evaluator needs a strong enough cage before measuring the animal. But the measurement is how the evaluator learns how strong the animal is. Most of the time, conservative engineering closes the gap: the sandbox is built with margins, the model stays within them, and the test produces useful evidence. The uncomfortable case is the frontier case, where the model is meaningfully more capable than expected.

In that case, the first honest measurement of the model’s strength could also be the first time the containment design is tested beyond its assumptions. The paradox has not been solved; it has been moved into the engineering quality of the sandbox and the governance quality of the decision to run the test.

📌

No perfect escape. The goal is not to eliminate the paradox, but to keep the safety margin ahead of the unknown capability.

You need a strong enough cage before measuring the model, but the measurement is how you learn its strength.

The decisions leaders and practitioners actually face

The paradox becomes practical when someone has to approve the test. A research lead, security architect, evaluation owner, or governance board must decide how much restraint to remove, what tools to provide, what containment to require, and what evidence is enough to proceed. These are not abstract philosophy questions. They determine whether the evaluation is meaningful and whether it is safe enough to run.

A weak evaluation may be politically comfortable because it is unlikely to produce frightening evidence. A strong evaluation may be scientifically valuable because it reveals the ceiling. The responsible path is neither to avoid hard tests nor to run them casually. It is to make the risk acceptance explicit, document the assumptions, and require controls that match the possibility that the model is better than expected.

The most important leadership habit is to separate two approvals: approval to learn, and approval to expose. A team may approve an unrestrained test inside a sealed lab while still refusing any external deployment or broad access until the results and traces have been reviewed.

What capability is being measured, and why can it not be measured with normal safeguards on?
Which safeguards are being relaxed, and for how long?
What tools, files, credentials, network paths, and targets can the model reach?
What stop conditions end the run immediately?
Who is allowed to review raw outputs, traces, and potentially harmful artifacts?
What result would trigger stronger containment, delayed release, or outside review?

How to live with the paradox

The paradox is unavoidable, but it is manageable. The practical answer is layered conservatism: start with weaker tasks, increase autonomy gradually, keep real assets out of reach, monitor the full trajectory, and assume the model may exceed the team’s estimate. Good evaluation design treats surprise as normal, not as an exception.

Progressive testing helps. A team can begin with desk-based review and tightly constrained tasks, then move to sealed simulations, then to longer agentic runs only after earlier stages show the containment is behaving as expected. Each stage should have explicit gates, logs, and rollback plans. The point is not to make the model harmless inside the test. The point is to make the consequences of its harmful behavior bounded and observable.

This also changes what counts as success. A successful dangerous-capability evaluation is not one where nothing alarming happens. It is one where the team learns something true without losing control of the environment. The best outcome may be an uncomfortable finding captured safely: evidence that the model can do more than expected, followed by a decision to strengthen controls before anyone is exposed to that capability.

Key takeaways

Measuring a model with safeguards on can show deployed safety behavior, but it may not reveal the model’s maximum capability.
An honest ceiling measurement often requires relaxing controls, which temporarily creates a less safe configuration of the model.
The sandbox is the main compensating control, so its design and operation carry the safety burden of the evaluation.
There is a circular trap: containment must be strong enough before the team knows exactly how strong the capability is.
Good practice manages the paradox through layered containment, progressive test stages, stop conditions, and full trajectory monitoring.
Leaders should approve learning separately from approving exposure; a lab finding is not a deployment decision.

Layered conservatism raises autonomy only after each earlier stage shows containment behaving as expected.

Glossary

Capability-measurement paradox: The problem that measuring a dangerous capability honestly can require relaxing the safeguards that normally prevent that capability from being expressed.
Frontier artificial intelligence model: A highly capable AI model near the leading edge of current performance, especially one whose limits are not yet fully understood.
Safety system: A control that limits, filters, refuses, interrupts, or otherwise constrains model behavior to reduce harmful outcomes.
Floor: A lower-bound measurement showing what a model does under restraint, not necessarily what it could do without restraint.
Ceiling: An estimate of the model’s maximum capability when the relevant restraints are removed or relaxed.
Sandbox: An isolated test environment designed to let risky behavior occur without reaching real systems, users, secrets, or networks.
Compensating control: A safeguard that makes a risky activity acceptable by reducing or containing the consequences of that activity.
Network egress: Outbound network access from a system to other machines, services, or the internet.
Trajectory monitoring: Observation of the model’s full sequence of prompts, reasoning steps, tool calls, outputs, and environmental changes during a run.