Timing Side Channels in Decoding: Speculative Acceptance and Bits-per-Query

Speculative decoding is a beautiful optimization: instead of generating one token at a time with the big, slow model, a small draft model guesses several tokens ahead, and the big model checks all the guesses in a single parallel step, keeping the ones it agrees with. When the draft guesses well, many tokens are accepted at once and generation flies; when the draft guesses badly, few are accepted and it crawls. The catch is that how well the draft guesses depends on how predictable the text is — and so the response time now carries a fingerprint of the content's predictability. An observer who times the tokens is reading a low-resolution shadow of what was generated, and that shadow can leak more than a serving team intends.

How speculative decoding couples content to time

In standard autoregressive decoding, each token costs one forward pass of the large model, so timing is roughly constant per token regardless of content. Speculative decoding, introduced by Leviathan and colleagues, breaks that uniformity for speed: a cheap draft model proposes k tokens, the large model verifies them in one pass, and it accepts the longest correct prefix of the draft. The number accepted per verification step — the acceptance rate — determines the speedup, and acceptance is high exactly when the draft and target agree, which is when the text is predictable.

This is the coupling. Predictable content (common phrasing, boilerplate, repetition) yields high acceptance and fast generation; surprising content (unusual names, high-entropy data, unexpected structure) yields low acceptance and slow generation. The per-token or inter-token latency therefore varies with the content's predictability, converting a property of the secret output into an observable timing signal. Nothing is broken; the optimization is behaving exactly as designed, and that design is what leaks.

The defensive consequence is that decoding timing is only safe to expose if it does not depend on content. Where it must not leak, the speedup mechanism must be neutralized for that traffic — constant-time-ish decoding — accepting the loss of the optimization's benefit as the price of confidentiality.

What the timing reveals

The leak is coarse but real: an observer of inter-token latency learns how predictable the generated text was, segment by segment. Fast bursts indicate predictable content; slow stretches indicate surprising content. This does not hand the attacker the text, but predictability is correlated with content in exploitable ways — a response that slows sharply at a particular position suggests a high-entropy token there (a name, a number, a rare word), which narrows the attacker's uncertainty about what was generated.

The signal strengthens when the attacker has partial control or knowledge. If an attacker can induce the model to generate a template with one secret field, the timing around that field reveals properties of the secret (its length, its predictability, whether it matches a common pattern). Combined with the batching couplings from the shared-serving threat model, decoding timing becomes one more observable that composes with others to reconstruct information about content the platform meant to keep private.

As with all side channels, severity is quantitative. A little content-predictability leaked per token, accumulated over a long response and many queries, can narrow a secret substantially. So the defense is not to dismiss the leak as coarse but to measure its bits-per-query and decide whether that rate, times the allowed query volume, is within an acceptable budget.

Quantifying bits-per-query

To manage the leak, measure it. Model the attacker's observable — the sequence of inter-token latencies — as a signal and estimate the mutual information between it and the secret content. For speculative decoding, a useful proxy is how much the acceptance-rate profile varies with content: if two very different secrets produce indistinguishable timing, the leak is small; if they produce reliably different timing, the leak is large. The bits-per-query figure then bounds how fast an attacker can narrow a secret and therefore whether rate limits can contain them.

The relationship below captures the accounting: the number of queries an attacker needs scales inversely with the per-query leak. Speculative decoding's leak per query depends on how content-dependent acceptance is for the traffic in question — boilerplate-heavy outputs leak little, high-entropy outputs leak more. This lets a defender target mitigation where it matters instead of paying constant-time costs everywhere.

The mitigations reduce the per-query leak toward zero and cap total queries. To reduce per-query leak, make timing content-independent: pad each token's emission to a fixed interval, or disable speculative decoding for sensitive traffic so every token costs the same constant time. To cap total leak, apply per-tenant rate limits. The combination keeps residual-leak times query-rate under a chosen budget, the same discipline as every other serving side channel.

Making timing content-independent

The core defense is constant-time-ish decoding: ensure the observable emission cadence does not depend on the content. The cleanest form is token-pacing — buffer generated tokens and emit them to the client at a fixed interval regardless of how fast they were actually produced, so the client sees a uniform stream whether the internal acceptance was high or low. This masks the speculative-decoding leak entirely for the emission channel at the cost of not passing the speedup through to latency.

A blunter option is to disable speculative decoding for sensitive traffic, making every token cost the same standard forward pass so there is no acceptance-driven variation to leak. This sacrifices the throughput benefit for those requests but restores the uniform timing of standard decoding. Which to choose depends on whether the platform can afford to buffer-and-pace (preserving internal speedup for cost while masking external timing) or must simply turn the optimization off.

Neither is free, which is why they should be applied by sensitivity rather than globally. Non-sensitive, public-output traffic can enjoy the full speculative speedup with exposed timing; sensitive traffic gets paced or non-speculative decoding. Making this an explicit per-traffic policy, rather than exposing speculative timing to everyone by default, is what prevents the leak from being universal.

Why this matters and how to deploy it

Any service that streams tokens to clients over a shared or observable channel is exposed, and the exposure matters most when outputs carry secrets — generated credentials, private data, or responses whose content a competitor would pay to infer. Agent backends that stream long, structured outputs are particularly affected, because a long response gives the attacker many tokens of timing signal and structured outputs often contain a few high-entropy secret fields whose timing stands out.

The synthesis is that speculative decoding, and any content-dependent optimization, trades confidentiality for speed through the timing channel, so the deployment rule is to expose speculative timing only for non-sensitive traffic and to pace or disable it for sensitive traffic, choosing the throughput cost against a measured bits-per-query budget. Leviathan and colleagues' speedup is real and worth having; the discipline is simply to remember that a speedup which depends on content is a signal about content, and to mask that signal wherever content must stay private.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

# DEFENSIVE PROBE — does decoding timing leak predictability?
function timing_leak(service, prompts):
    rows = []
    for p in prompts:
        toks, lat = service.generate_timed(p)
        rows.append((segment_entropy(toks), inter_token_latency(lat)))
    r = correlation([e for e,_ in rows], [l for _,l in rows])
    if abs(r) > LEAK_THRESHOLD:
        flag("timing correlates with content entropy", r)
    # Strong correlation == exploitable decoding-timing channel; pace it.

decoding_timing_policy:
  speculative_decoding:
    sensitive_traffic: disabled_or_paced
    public_traffic: enabled
  emission:
    pacing: fixed_interval_for_sensitive
  metering:
    per_tenant_rate_limit: enforced
    leakage_budget_bits: defined
  classification:
    secret_field_outputs: always_paced

Glossary

Speculative decoding

Generating faster by having a small draft model propose tokens the large model verifies in parallel.

Draft model

A small, fast model that proposes candidate tokens for the large target model to verify.

Acceptance rate

The fraction of drafted tokens the target model accepts, which drives the speedup and depends on predictability.

Inter-token latency

The time between successive emitted tokens, made content-dependent by speculative decoding.

Constant-time-ish serving

Making observable timing independent of content, typically by pacing token emission at a fixed interval.

Token pacing

Buffering generated tokens and emitting them at a fixed interval to mask internal timing variation.

Bits-per-query

The information a single query's timing profile reveals about the secret content.

Entropy

A measure of unpredictability; high-entropy tokens accept poorly and generate slowly, standing out in timing.

References

1. Leviathan et al., Fast Inference from Transformers via Speculative Decoding (arXiv 2211.17192)
2. Kocher, Timing Attacks on Implementations of Diffie-Hellman, RSA, DSS, and Other Systems (CRYPTO 1996)
3. Kwon et al., Efficient Memory Management for LLM Serving with PagedAttention (arXiv 2309.06180)
4. Shumailov et al., Sponge Examples: Energy-Latency Attacks on Neural Networks (arXiv 2006.03463)
5. NIST AI 100-2 e2023, Adversarial Machine Learning: A Taxonomy and Terminology