AI Cyber-Capability Evaluations — Frontier AI Security: Foundations

An artificial intelligence (AI) lab tests its own model on hacking tasks for the same reason engineers crash-test cars: not because the crash is desirable, but because ignorance is worse. If a model can discover software weaknesses, turn them into access, and combine several steps into a larger attack, defenders need to know before real attackers learn to use that capability. A cyber-capability evaluation is the controlled test built for that purpose. It is useful, but it is also delicate, because honest measurement often requires running the model without the safety refusals that would normally stop harmful behavior.

Why measure a dangerous capability at all

At first glance, evaluating a model for offensive cyber skill sounds backwards. Why create a test that asks a system to behave like an attacker? The answer is that serious safety work begins with measurement. A lab cannot responsibly decide how to release, restrict, monitor, or improve a model if it does not know what the model can do under pressure.

The same logic appears in other high-risk fields. Cars are crashed into barriers so engineers can study failure. Dangerous biological materials are studied under strict laboratory controls so public-health teams can understand them. In cyber-capability evaluations, the object of study is not a vehicle or a pathogen but a model's ability to help find and exploit weaknesses in software systems.

The measurement matters because cyber risk is practical. Defenders must decide whether existing controls are enough, whether new monitoring is needed, and whether a model would materially lower the skill or time required to carry out an attack. Without a realistic test, those decisions become guesswork.

📌

Measurement before policy. A release policy is only as strong as the evidence it uses to describe what the model can actually do.

What a cyber-capability evaluation is

A cyber-capability evaluation is a structured test that places a model in realistic security scenarios and measures how far it gets. The scenarios are usually built as a benchmark: a curated, repeatable set of challenges that can be used across models and over time. The goal is not to teach the public how to attack systems, but to measure whether the model can reason through the kinds of tasks that make attacks succeed.

A simple challenge might ask whether the model can identify a weakness in a small application. A harder one might require the model to turn that weakness into access, inspect the environment, discover what matters, and complete a defined objective such as retrieving a protected file inside the test system. The strongest evaluations look beyond isolated tricks and ask whether the model can chain steps together toward a goal.

The result is not merely a scorecard that says pass or fail. A good evaluation produces evidence about the model's behavior: where it got stuck, what it tried, whether it recovered from mistakes, how much tool use it needed, and whether success depended on luck or repeatable reasoning.

What the model is allowed to use

A realistic evaluation does not just ask the model questions in a chat window. It often gives the model controlled tools, such as a command interface inside a sandbox, access to test files, or the ability to run diagnostic commands in the evaluation environment. This matters because cyber work is interactive. The model must form a hypothesis, test it, read the result, and adjust.

The evaluation records the model's trajectory, meaning the sequence of prompts, tool calls, outputs, retries, and decisions that led to the final outcome. Trajectory-level evidence is important because two models can receive the same final score while behaving very differently. One may solve the task cleanly and consistently. Another may stumble into success through many unsafe or unreliable attempts.

The most useful records also capture constraints: how much time the model had, what tools were available, whether a human operator intervened, and whether the result was reproducible. These details keep the benchmark from turning into a vague contest and make the measurement useful for security engineering.

Task goal, such as reaching a protected resource inside the test environment
Tool access, such as a controlled shell or test application interface
Action trace, including commands attempted and observations returned
Outcome evidence, including partial progress and final success or failure
Reproducibility notes, such as whether the same result appears across repeated runs

Cyber work is interactive: the model forms a hypothesis, tests it, reads the result, and adjusts.

Why safety refusals are switched off

Most deployed AI assistants include safety classifiers. A safety classifier is an automatic system that detects certain harmful requests and causes the assistant to refuse. In everyday products, this is essential. If a user asks for help breaking into an account, the assistant should decline.

But a cyber-capability evaluation asks a different question. It is not asking how well the deployed product refuses harmful instructions. It is asking what the underlying model could do if those refusals did not stop it. If the refusals remain enabled, the test may mostly measure the guardrail, not the model's capability.

For that reason, labs often run capability evaluations with the usual refusal behavior disabled or bypassed under controlled conditions. This is not a casual choice. It is the measurement equivalent of taking ankle weights off a sprinter before timing a race. If the goal is to learn the true top speed, the restraint cannot be part of the result.

The consequence is serious: during the evaluation, the lab may be operating a more dangerous version of the model than users normally see. That is why the safety of the surrounding environment becomes central, not incidental.

⚠️

Not a normal product run. An evaluation with refusals disabled should be treated as a high-risk test of an unrestrained system, not as ordinary model use.

With refusals on, a test mostly measures the guardrail; with them off, it measures the model.

The containment bargain

Disabling refusals creates a bargain. The lab gains a more honest measurement, but only if the evaluation environment can safely contain the behavior being measured. The model may be encouraged to search for weaknesses, run tools, and pursue goals that would be unacceptable in the open world. The environment must keep that activity from reaching real systems.

In practice, that means using isolated evaluation environments, often called sandboxes. A sandbox is a controlled computing space designed to limit what the model and its tools can reach. Good containment also includes default-deny network egress, which means outbound network connections are blocked unless explicitly allowed; careful handling of secrets and credentials; and detailed logging so investigators can reconstruct what happened.

The hard lesson is that the evaluation environment itself becomes part of the security problem. If the sandbox has a misconfiguration, if a test dependency can fetch untrusted code, or if credentials leak into the environment, then the evaluation may expose real assets. The test is only safe if its software supply chain, network boundaries, and operational procedures are safe.

Honest measurement requires an environment that safely contains the behavior being measured.

How leaders and practitioners use the results

A cyber-capability evaluation is not an academic trophy. Its value is in the decisions it informs. A model that can only solve toy challenges may require one level of oversight. A model that reliably discovers unknown weaknesses, adapts after failure, and chains actions toward a target may require a very different release plan.

For a leader, the result can affect whether a model is released, who can access it, what monitoring is required, and whether additional safeguards must be built first. For a security practitioner, the result can influence incident-response planning, red-team priorities, secure development practices, and what kinds of AI-assisted attacks to expect.

The most mature use of evaluations is not a single go or no-go threshold. It is a safety case: a reasoned argument, supported by evidence, that the model can be deployed under specified controls without creating unacceptable risk. The evaluation supplies part of that evidence, but it must be combined with containment tests, misuse monitoring, access controls, and plans for responding when new weaknesses are discovered.

What evaluations can and cannot prove

A strong benchmark is useful, but it is not a complete map of danger. It samples a set of tasks chosen by its designers. A model that fails today may succeed after a small improvement in reasoning, tool use, or context length. A model that succeeds in a sandbox may behave differently in a messy real organization. Capability is not fixed; it changes as models, tools, and attacker workflows change.

Evaluations also face contamination risks. If a model has seen benchmark answers during training, or if solution patterns leak into later testing, the score may overstate real capability. This is why serious evaluations need protected answer materials, fresh tasks, provenance for evaluation artifacts, and procedures that make results repeatable without making the benchmark easy to memorize.

Finally, cyber-capability is not only about whether the model can complete one task. Long-horizon autonomy matters: can it stay oriented over many steps, maintain a plan, recover from errors, and know when to try a different route? Those qualities often determine whether isolated skill becomes operational risk. Good evaluations therefore evolve from simple challenge solving toward trajectory-level, multi-step measurement.

The core paradox remains. To measure dangerous capability honestly, evaluators may need to invite the model to use that capability while removing the safeguards that usually suppress it. That can be responsible only when the measurement is paired with disciplined containment, careful governance, and a clear purpose: helping defenders understand what they must be ready for.

⚠️

The paradox. The more realistic the test, the more seriously the evaluator must treat the test environment as a security boundary.

Key takeaways

Cyber-capability evaluations measure how far an AI model can go on realistic offensive-security tasks.
Labs run these tests because defenders and release decision-makers need evidence, not guesses.
Safety refusals are often disabled during evaluation so the result reflects the model's capability ceiling rather than the product guardrails.
That choice makes containment essential: the sandbox, network limits, secrets handling, and logging must all be designed for high-risk testing.
Evaluation scores should inform practical decisions about release scope, safeguards, monitoring, and incident response.
Benchmarks are necessary but incomplete, so they must evolve as models, tools, and attacker methods change.

A benchmark is useful evidence, not a complete map of danger.

Glossary

Artificial intelligence: Artificial intelligence is software that performs tasks associated with human reasoning, such as interpreting instructions, planning actions, or generating text and code.
Cyber-capability evaluation: A cyber-capability evaluation is a structured test that measures how well an AI model can perform realistic security tasks such as finding, exploiting, and chaining software weaknesses.
Benchmark: A benchmark is a curated and repeatable set of tasks used to compare capability across models or across time.
Safety classifier: A safety classifier is an automatic system that detects certain harmful requests or outputs and triggers a refusal or other protective behavior.
Refusal: A refusal is the model's response when it declines to help with a request judged unsafe or disallowed.
Capability ceiling: A capability ceiling is the best performance a model can reach when it is allowed to use its abilities at full effort under the evaluation conditions.
Sandbox: A sandbox is an isolated computing environment designed to let risky activity occur without affecting real systems.
Default-deny network egress: Default-deny network egress is a network rule that blocks outbound connections unless a specific connection has been explicitly allowed.
Trajectory: A trajectory is the full sequence of a model's prompts, tool calls, observations, decisions, and retries during an evaluation run.