Stealing the Last Layer: Recovering a Production Model's Final Projection from Logits

It is tempting to assume that a model too large to steal outright keeps all of its structure secret behind an API. Carlini and colleagues showed otherwise for a specific, high-value piece: the final projection layer — the linear map from the model's hidden state to its output logits — can be recovered from the API alone, along with the hidden dimension it projects from. The reason is linear algebra, not a breach. The output logits are a linear function of a much smaller hidden vector, so every response is a sample from a low-dimensional subspace, and enough samples pin down that subspace and the map that generated it. This article develops the mechanism and, more importantly, the disclosure-side settings that govern whether the leak is open or closed.

The last layer is a low-rank linear map

A language model computes a hidden state of modest dimension for each position, then applies a final linear projection to produce a logit for every token in a large vocabulary. That projection is a single matrix multiplying the hidden vector, so the logit vector — however long — is a linear image of a vector whose dimension is the hidden size. Because the vocabulary is far larger than the hidden dimension, the projection is low-rank: all logit vectors the model can emit lie within a subspace whose dimension equals the hidden size.

This is the entire vulnerability in one sentence: the output space looks high-dimensional but is constrained to a low-dimensional subspace fixed by the last layer. An attacker who collects many logit vectors is sampling points from that subspace, and a subspace is determined by enough points in it. The rank of the collected data reveals the hidden dimension, and the directions it spans reveal the column space of the projection matrix.

For a defender, the crucial recognition is that this leak is structural, not implementation-specific. Any API that exposes the full logit vector — or enough of it — is exposing samples from this subspace. The control is therefore about what the API returns, exactly the disclosure lever the query-access model identifies.

Recovering the hidden dimension

The first thing an attacker recovers is the hidden dimension itself, and it falls out of a singular-value decomposition. Collect logit vectors for many varied queries, stack them into a matrix, and compute its singular values. Because the logits lie in a subspace of dimension equal to the hidden size, the matrix has that many large singular values and the rest are essentially zero — a sharp drop at the hidden dimension. Reading off where the singular values collapse gives the model's hidden size directly.

This is a clean, quantifiable measurement rather than a guess. The singular-value spectrum shows a plateau of significant values followed by a cliff, and the location of the cliff is the hidden dimension. Noise in the outputs blurs the cliff but does not hide it unless the noise is large enough to matter, which is precisely why rounding and noising outputs is a defense: they raise the floor the small singular values sit on and obscure the transition.

Recovering the hidden dimension is valuable on its own — it reveals a closely-guarded architectural fact and narrows the space of what the model could be — and it is the prerequisite for recovering the projection. The defense is to deny the clean logit vectors the decomposition needs.

Recovering the projection up to symmetry

With the hidden dimension known, the attacker can recover the projection matrix — not uniquely, but up to a symmetry. The column space spanned by the collected logits is the column space of the true projection, so a decomposition yields a matrix that equals the real one composed with an unknown invertible transform on the hidden side. This is inherent: the model's behavior is unchanged if you apply any invertible map to the hidden state and its inverse to the projection, so the API cannot distinguish these equivalent factorizations.

What the attacker obtains is therefore a functionally faithful reconstruction of the last layer's action, up to that hidden-side ambiguity. That is enough to reproduce the mapping from hidden states to outputs, to learn the exact output-layer geometry, and to bootstrap further analysis — even though it is not the literal stored matrix. The pipeline is short: query broadly, stack the logit responses, decompose to find the dimension, then solve for the projection up to the symmetry.

The defensive lesson repeats: every stage consumes clean logit vectors from the low-rank subspace. Deny or corrupt those vectors and the pipeline stalls at its first step. The recovery is elegant precisely because it needs nothing but honest, rich responses.

How API design leaks the signal

Whether this attack is possible is decided by API design. An API that returns the complete logit vector hands the attacker subspace samples directly. But even APIs that do not expose full logits can leak the same signal through convenience features: a logit-bias parameter that lets a caller nudge specific tokens can be used to probe the projection one direction at a time, and a top-k interface combined with such biasing can be steered to read out more of the vector than intended. The leak rides on features meant for legitimate control.

The contrast between a leaky and a hardened API is stark. A leaky configuration returns full or bias-probeable logits with generous query budgets; a hardened one returns only what the application needs — a label or a small, rounded top-k without controllable bias — and caps query volume. The same model behind the two configurations has very different confidentiality, which is the whole point: the model did not change, the disclosure did.

This is why the defense is a design review of the response surface, not a patch to the model. Enumerate every way a caller can influence or read the output vector — full logits, logit bias, large top-k, high-precision scores — and close the ones that reconstruct the subspace.

What it does and does not yield

It is important to bound the result honestly. Recovering the last layer and the hidden dimension does not steal the whole model — the many layers that compute the hidden state remain hidden, and the recovered projection is only up to a symmetry. What it does yield is a genuinely sensitive architectural fact (the hidden size), a faithful reconstruction of the output layer's behavior, and a foothold that lowers the cost of further extraction and of building a competing surrogate. It is a real confidentiality loss, not a total compromise.

The value to an attacker is therefore both direct and instrumental. Directly, the hidden dimension and output geometry are proprietary details a provider would prefer to keep secret. Instrumentally, knowing them constrains the model and can accelerate other attacks, including behavioral surrogate construction and transfer. Sizing the defense means recognizing this as a meaningful partial leak worth closing, not dismissing it because it is not a full weight theft.

The proportionate response is disclosure control commensurate with the sensitivity of the leaked facts. For many providers the hidden dimension and output-layer geometry are worth protecting, which justifies returning less and monitoring for the broad, structured querying the recovery requires.

Defenses that follow from the mechanism

The mechanism dictates the defenses, all on the disclosure side. First, minimize what the API returns: prefer labels or a small top-k over full logit vectors, since subspace recovery needs the vectors. Second, corrupt the signal that remains: round or add calibrated noise to scores so the singular-value cliff blurs and the projection cannot be solved cleanly. Third, remove reconstruction features: disable or tightly constrain logit-bias and high-precision score outputs that let a caller read the projection direction by direction. Fourth, throttle and detect: the recovery needs broad, structured querying to sample the subspace, a pattern anomaly detection can flag.

These compose into a response-surface hardening that leaves legitimate use intact while denying the subspace. The key insight is that the attack is fragile to disclosure reduction — it depends on clean, rich outputs — so modest reductions in what is returned raise its cost sharply. Unlike model-level attacks, there is no need to retrain; the fix lives entirely in the serving layer's output policy.

The measured posture is to decide, per deployment, the minimum output richness the application truly needs, deliver only that with rounding, strip reconstruction-enabling features, and watch for subspace-sampling query patterns. That closes the last-layer leak at its source.

Limits and honest framing

The attack and its defenses both have limits. The recovery yields the last layer up to a symmetry and the hidden dimension, not the full model, so the confidentiality loss is real but partial; over-stating it as full theft misleads risk assessment. On the defense side, output noising and rounding cost utility for callers who genuinely need precise scores, and removing logit-bias features can break legitimate control use cases — so hardening is a trade, not free, and must be scoped to the application's real needs.

There are also detection caveats. The subspace-sampling pattern is distinctive but can be spread across accounts and time to evade anomaly detection, which is why disclosure minimization and score corruption — controls that reduce leakage per response regardless of who queries — must carry the load rather than detection alone. Detection is a supplement to a fundamentally disclosure-side fix.

Held honestly, the last-layer result is a precise example of the query-access thesis: a specific, high-value structural fact leaks not through a breach but through what the API returns, and the fix is to return less and corrupt what remains. The mechanism names exactly which output settings govern the exposure, turning a surprising attack into a manageable, disclosure-side control.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

# DEFENSIVE / MOCK ONLY — audits configuration, does not recover any layer
function audit_response_surface(api_config):
    findings = []
    if api_config.returns_full_logits:
        findings.append('full logits: subspace directly sampleable')
    if api_config.logit_bias_enabled and not api_config.bias_constrained:
        findings.append('logit bias: projection probeable per direction')
    if api_config.score_precision > MAX_SAFE_PRECISION:
        findings.append('high-precision scores: clean SVD possible')
    if api_config.per_client_query_budget > SUBSPACE_SAMPLE_T:
        findings.append('budget allows dense subspace sampling')
    return AuditReport(findings, recommend='minimize disclosure; round; strip bias')

Glossary

Final projection layer

The linear matrix mapping a model's hidden state to its output logits over the vocabulary.

Hidden dimension

The size of the model's internal hidden state, equal to the rank of the logit subspace.

Low-rank subspace

The constrained region logit vectors occupy because they are linear images of a smaller hidden vector.

Singular-value decomposition

A factorization whose significant-value count reveals the rank, hence the hidden dimension, of stacked logits.

Recovery up to symmetry

Reconstruction of the projection composed with an unknown invertible hidden-side transform the API cannot resolve.

Logit bias

An API feature letting a caller nudge specific token scores, abusable to probe the projection directionally.

Disclosure control

Serving-layer policy on output richness, precision, and features that governs subspace leakage.

References

1. Carlini et al., Stealing Part of a Production Language Model (arXiv 2403.06634)
2. Tramèr et al., Stealing Machine Learning Models via Prediction APIs (USENIX Security 2016)
3. NIST AI 100-2 e2023 — Adversarial Machine Learning: A Taxonomy and Terminology
4. Shokri et al., Membership Inference Attacks Against Machine Learning Models (IEEE S&P 2017)
5. Carlini et al., Extracting Training Data from Large Language Models (arXiv 2012.07805)