The Query-Access Threat Model: What a Black-Box Attacker Can and Cannot Learn

A model served behind an API is often treated as confidential by construction: the weights sit on the provider's servers, the client sees only outputs, and secrecy is assumed. That assumption is wrong in a specific, measurable way. Tramèr and colleagues showed that prediction APIs leak enough to reconstruct model behavior, and later work extended this to recovering structural pieces of production language models. The reason is simple: every response carries information about the function that produced it, and an attacker who issues enough well-chosen queries accumulates that information. This article defines the query-access threat model — what an attacker with only input-output access can pursue, what the API's disclosure level determines, and the information ceiling that governs it all — so a defender can see exactly which knob controls the leak.

The boundary is the response, not the network

The confidentiality boundary of an API-served model is not the network perimeter; it is the content of each response. Whatever the model returns — a label, a probability vector, a set of logits — is a measurement of the underlying function at the queried point, and measurements accumulate into knowledge. NIST's adversarial-machine-learning taxonomy frames this as the adversary's knowledge and capability: a black-box attacker has query capability and observes outputs, and the security question is what that observation channel reveals.

This reframing is the key to reasoning about extraction. The defender does not get to decide whether responses carry information — they always do — but does get to decide how much, by choosing what the API returns and how often. The threat model is therefore parameterized by two things the provider controls: the disclosure level per response and the query budget per client. Both are levers, and both are quantifiable.

The rest of this piece develops the model along those axes: what an attacker can target, how disclosure level changes the leak, and the information-theoretic ceiling that ties total leakage to the number and richness of responses. The goal is a defender who can point to the specific setting that governs a given exposure.

A taxonomy of extraction targets

Extraction is not one attack but a family sorted by what the attacker wants to steal. The first target is parameters: the actual weights, or a functionally equivalent copy that reproduces the model's outputs. The second is architecture: structural facts like hidden dimensions, which can leak even when full weights do not. The third is the decision boundary: a surrogate that mimics the model's input-output behavior closely enough to substitute for it or to craft transferable attacks. The fourth is training data: specific examples the model memorized, a confidentiality and privacy target distinct from the model itself.

These targets differ in difficulty and in what disclosure level enables them. A functionally equivalent copy of a simple model can be solved for from enough input-output pairs; a structural fact like an embedding dimension can be recovered from rich outputs even for a large model; a decision-boundary surrogate needs only labels but many of them; and training-data extraction depends on memorization rather than on the query channel's richness. Naming them separately prevents the common error of treating all extraction as equally hard.

For the defender, the taxonomy is a checklist of exposures, each with its own dominant lever. Parameter and boundary theft are throttled by limiting query volume and output richness; architectural leakage is throttled by not returning the rich outputs that reveal structure; data extraction is addressed at training time. One boundary, several distinct leaks.

The disclosure spectrum

How much a single response reveals runs along a spectrum of API disclosure. At the low end, a label-only API returns just the predicted class or a single answer, leaking the least per query — an attacker learns only which side of the boundary the input fell on. In the middle, a top-k probability API returns the most likely outputs with scores, revealing far more about the function's shape near the query. At the high end, returning full logits or complete probability vectors exposes the model's raw pre-softmax values, the richest possible per-query measurement and the easiest to invert.

The spectrum matters because per-query information is the multiplier on the whole attack. A label-only channel forces the attacker to spend many queries to learn what a single logit-rich response would give; a logit-rich channel can leak a structural fact in comparatively few queries. This is exactly why recovering a production model's final projection became feasible from logit-style outputs: the richness of the response, not a network breach, carried the signal. Every increment of disclosure moves the attacker's cost down.

The defensive reading is direct: returning less per response is the highest-leverage control, because it raises the query cost of every extraction target at once. Truncating to top-k, rounding scores, or returning labels where possible each shifts the deployment down the spectrum toward less leakage.

What is learnable, and what is not

Not everything is extractable, and separating the two is essential to avoid both complacency and paranoia. Functionally equivalent behavior on the queried region is broadly learnable given enough responses — the attacker can build a surrogate that agrees with the target where it was probed. Certain structural facts, like output-layer dimensions, are learnable from rich responses. But an exact bit-for-bit copy of a large model's weights is generally not recoverable from the API alone, because many weight configurations produce the same outputs and the query channel does not disambiguate them.

The practical boundary is between behavioral and exact extraction. An attacker can usually obtain a good behavioral surrogate — sufficient to substitute for the service or to mount transferable attacks — but rarely the precise internal parameters of a large model through queries alone. This distinction should calibrate defense: the realistic threat is a cheaper competitor model or a transfer-attack surrogate, not necessarily a perfect weight theft, and defenses should be sized to the realistic threat.

The learnable set also grows with disclosure and query budget, which is why those are the levers. What is out of reach at label-only, low-budget access can come into reach at logit-rich, high-budget access. The defender's job is to keep the valuable targets outside the learnable set for a realistic attacker.

The information-theoretic ceiling

The cleanest way to reason about the whole model is a budget: total extractable information is bounded by the number of queries times the information per response. This ceiling is not a precise attack cost but a governing relationship — halve the per-response information or the query budget and you halve the ceiling on what can be stolen. It explains why the two provider-controlled levers, disclosure level and query rate, are the fundamental controls: they are the two factors in the product.

The framing also clarifies why extraction is a measurement problem in the data-science sense. Each query is a noisy measurement of the target function; the attacker is an estimator accumulating measurements toward a model of it; and the defender's counter is to make each measurement carry fewer bits or to allow fewer of them. Rounding outputs adds noise to each measurement; rate-limiting reduces their count; both shrink the estimator's achievable precision.

Read as an inequality, the ceiling turns defense into arithmetic. A provider can estimate the per-response information of its disclosure level, set a query budget, and thereby bound the leakage a single client can achieve — a far more defensible posture than assuming the network boundary makes the model secret.

Defenses that follow from the model

Because leakage is the product of disclosure and query budget, the defenses are the factors of that product plus detection. Minimize disclosure: return the least informative response the application tolerates, and round or truncate scores to strip bits. Constrain the query budget: rate-limit per client and per account, and detect the systematic, boundary-probing query patterns that extraction requires, since those patterns differ from ordinary use. And monitor: an attacker building a surrogate issues characteristic sweeps that anomaly detection can flag before the budget is spent.

These controls compose. Lower disclosure raises the queries needed for any target; a query budget caps how many are available; detection shortens the window before the budget is cut off. None makes the model perfectly secret — responses always leak something — but together they hold the extractable information below what a realistic attacker can turn into a valuable copy. The defense is economic and quantifiable, matching the threat model's own logic.

The measured way to run this is to set disclosure and budget deliberately against the value of the model and the sensitivity of its training data, then monitor for the query signatures of extraction. That is a defensible posture; assuming the API is a wall is not.

Limits and honest framing

The model has limits worth stating. The information ceiling is a bound, not an exact cost; real attacks may fall well short of it or exploit structure to approach it faster, so it guides rather than predicts. Disclosure minimization trades against utility — many legitimate uses need probabilities or logits — so returning less is not free. And query-pattern detection faces the usual evasion: an attacker can spread queries across accounts and time to look ordinary, which is why budget and disclosure controls, not detection alone, must carry the load.

There are also targets the query channel does not govern. Training-data extraction depends on what the model memorized during training, not on how rich the API is, so it is addressed at training time by memorization-limiting techniques rather than by disclosure controls — a separate exposure this threat model flags but does not close. Recognizing that boundary prevents over-trusting API-side defenses for a training-time problem.

Held honestly, the query-access model gives the defender exactly what a threat model should: a clear enumeration of targets, the two levers that govern most of them, an information ceiling that turns defense into arithmetic, and an honest marker of where the model stops — at the training-time leaks a later analysis must handle. Return less, allow fewer, watch for sweeps, and address memorization separately.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

# DEFENSIVE / MOCK ONLY — bounds and monitors, extracts nothing
function govern(client, request):
    bits = bits_per_response(DISCLOSURE_LEVEL)     # label < top-k < logits
    client.spent_bits += bits
    if client.query_count > QUERY_BUDGET:
        return DENY(reason='query budget exceeded')
    if boundary_probe_score(client.recent_queries) > SWEEP_T:
        return REVIEW(reason='extraction-sweep pattern')
    if client.spent_bits > LEAKAGE_CEILING:
        return THROTTLE(reason='leakage ceiling reached')
    return ALLOW

Glossary

Query-access (black-box)

An attacker capability limited to issuing inputs and observing the model's outputs, with no weight access.

Disclosure level

How much each API response reveals — a label, top-k probabilities, or full logits.

Functionally equivalent extraction

Recovering a surrogate that reproduces a target model's outputs, without recovering its exact weights.

Decision-boundary surrogate

A model trained to mimic a target's input-output behavior, usable for substitution or transferable attacks.

Logits

A model's raw pre-softmax output values, the richest per-query measurement of its function.

Information ceiling

The bound on extractable information as query budget times information per response.

Query budget

The number of responses a client is allowed, one of the two provider-controlled leakage levers.

References

1. Tramèr et al., Stealing Machine Learning Models via Prediction APIs (USENIX Security 2016)
2. Carlini et al., Stealing Part of a Production Language Model (arXiv 2403.06634)
3. Shokri et al., Membership Inference Attacks Against Machine Learning Models (IEEE S&P 2017)
4. Carlini et al., Extracting Training Data from Large Language Models (arXiv 2012.07805)
5. NIST AI 100-2 e2023 — Adversarial Machine Learning: A Taxonomy and Terminology