The Shared-Serving Threat Model: Co-Tenancy, Batching, and the Attacker's Observables

A model-serving stack is a marvel of systems engineering: it packs many users onto shared accelerators, batches their requests to keep the hardware busy, caches repeated computation so it is never done twice, and speculatively runs ahead to shave latency. Every one of those tricks works by sharing state or time across requests — and anything shared across a trust boundary is a potential side channel. The attacker in this model is not breaking the model or the prompt; they are a paying co-tenant watching the clock. When your request makes their request faster or slower, they learn something about you, and the discipline of side-channel analysis, developed decades ago for cryptographic implementations by Kocher and others, applies directly to the shared machine your prompts run on.

Why sharing is the vulnerability

Confidentiality between tenants requires isolation, but isolation is expensive, so serving systems trade it away for efficiency. When two tenants share an accelerator, a cache, or a batch, the resource each consumes affects the other's observable performance. The attacker does not need to read another tenant's memory; they need only to measure their own latency and throughput precisely and infer, from how those change, what the victim is doing. This is the essence of a timing side channel: secret-dependent behavior becomes secret-dependent timing.

The leak exists because the serving stack's optimizations are input-dependent. A cache hit is faster than a miss, and whether you hit depends on what others have sent. A batch's latency depends on the other requests batched with it. Speculative decoding's speedup depends on how predictable the text is. Each of these couples one tenant's secret inputs to another tenant's measurements, and the coupling is exactly what an attacker exploits.

The defensive framing that follows is that shared state across a trust boundary must be treated as an information channel and either eliminated (partition per tenant), masked (make timing independent of secrets), or metered (limit how much an attacker can observe). Which of these to apply, and where, is the substance of the rest of this threat model.

The attacker's observables

A co-tenant attacker cannot see another user's prompt or response, but they can measure a rich set of timing and ordering signals about their own requests, and those signals are modulated by the victim's traffic. The primary observables are latency (how long a request takes end to end and token by token), throughput (how many tokens per second they achieve under contention), and ordering (which of their requests complete first when batched against others). Secondary observables include queueing delay, time-to-first-token, and inter-token latency.

What makes these powerful is precision and repetition. An attacker can issue many probe requests, measure timing to microseconds, and average away noise, extracting a stable signal from a noisy channel — the same statistical approach that makes cryptographic timing attacks practical. The relevant question is not whether a leak exists but how many bits per query it carries, because that determines how quickly an attacker can recover a secret such as whether a particular prompt prefix is cached.

The defender's counter is to make these observables independent of other tenants' secrets: constant-work scheduling, padding timing to fixed quanta, and per-tenant isolation so contention does not cross the boundary. Each reduces the mutual information between the victim's inputs and the attacker's measurements toward zero, at a throughput cost that must be chosen deliberately.

Caching: the sharpest channel

Caching is the most consequential shared state because its whole purpose is to make repeated work fast, which means a cache hit is directly observable as reduced latency. In model serving, prompt-prefix caching (reusing the key-value cache for a shared prompt prefix) is a common optimization: if two requests share a prefix, the second reuses the first's computation. Across a tenant boundary, this turns the cache into a membership oracle — an attacker can test whether a specific prefix is already cached (implying some other tenant recently sent it) by measuring whether their request is suspiciously fast.

This is a classic shared-cache side channel adapted to model serving, and it is dangerous because prompts often contain sensitive content: system prompts, proprietary templates, personal data. An attacker who can confirm that a particular prefix was recently processed learns about other tenants' activity without ever seeing their data. The leak is quantifiable — each timed probe yields roughly one bit (cached or not) about a chosen prefix — so with enough probes an attacker can search a space of candidate prefixes.

The defense is cache isolation: partition caches per tenant so a hit can only reflect the same tenant's prior requests, never another's. Where full partitioning is too costly, the fallbacks are to disable cross-tenant prefix sharing for sensitive prompts and to add timing noise so a single probe does not cleanly reveal hit versus miss — though noise only slows an averaging attacker rather than stopping them.

Batching and speculative decoding

Dynamic batching groups concurrent requests to use the accelerator efficiently, but a batch couples its members' timing: the batch proceeds at the pace of its slowest member, and completion ordering reflects the relative work of the requests in it. An attacker who submits requests alongside a victim can, by measuring how their own requests are delayed or reordered, infer coarse properties of the victim's load — request size, arrival pattern, or priority. Kwon and colleagues' PagedAttention and similar systems make batching efficient precisely by sharing memory and scheduling across requests, which is exactly the shared state that couples them.

Speculative decoding, introduced by Leviathan and colleagues, speeds generation by having a small draft model propose tokens that the large model verifies in parallel; the speedup depends on how many drafted tokens are accepted, which depends on how predictable the text is. This makes inter-token latency data-dependent: predictable continuations run faster than surprising ones. Across a boundary, or even within a response an attacker can partially influence, this couples timing to content, leaking information about how expected the generated text is.

The defenses trade efficiency for isolation: schedule sensitive tenants in isolation rather than co-batched with untrusted traffic, and where timing must not depend on content, disable or pad speculative decoding for sensitive requests so inter-token latency is constant. As always, the mitigation is a deliberate throughput sacrifice, not a free fix.

Quantifying the leak

Side-channel severity is not binary; it is measured in information. The right question for any shared-serving optimization is how many bits about a victim secret an attacker extracts per query, because that fixes how many queries they need to recover the secret and therefore whether rate limits and monitoring can contain them. A channel that leaks a tiny fraction of a bit per query under heavy noise may be tolerable; one that leaks a clean bit per probe (as an unmitigated cache oracle does) is not.

This framing turns defense into a measurable engineering target. Model the attacker's observable as a random variable, estimate the mutual information between it and the victim secret, and drive that mutual information down with masking and partitioning until the residual leak times the achievable query rate is below a tolerable threshold. It also clarifies why noise alone is weak: adding noise lowers per-query information but an attacker recovers it by averaging over more queries, so noise must be paired with rate limiting that caps the total queries — and thus total bits — available.

The synthesis is that shared-serving defense is a quantitative isolation problem: measure the channel, reduce its per-query capacity by masking or partitioning, and cap its total capacity by metering, choosing the throughput cost knowingly rather than discovering the leak in production.

Why this matters for agents and multi-tenant platforms

Any platform that serves many customers from shared model infrastructure — an API provider, an internal multi-team gateway, an agent backend handling many users — inherits this threat model. The stakes rise for agents because their prompts often carry credentials, private context, and proprietary system instructions, exactly the sensitive prefixes a cache oracle can probe for. A co-tenant who confirms the presence of a specific system-prompt prefix or a specific user identifier learns something the platform promised to keep isolated.

The synthesis is that inference serving must treat performance and confidentiality as an explicit trade-off decided per tenant. The controls are known — cache partitioning, isolated scheduling for sensitive tenants, constant-time-ish serving where timing must not depend on content, and per-tenant quotas that cap total observable queries — and each costs throughput. Kocher's decades-old lesson that shared timing is a channel, combined with the systems mechanics of Kwon, Leviathan, Shumailov, and colleagues, says the same thing for model serving: measure what your optimizations share, and never let an untrusted co-tenant observe more than your leakage budget allows.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

# DEFENSIVE PROBE — does our cache leak hit/miss via timing?
function cache_leak_probe(service, prefix):
    cold = median(time(service.request(prefix)) for _ in warm_miss_setup())
    warm = median(time(service.request(prefix)) for _ in range(N))  # cached
    separation = (cold - warm) / noise_std(service)
    if separation > DETECTABLE:
        flag("cache timing distinguishes hit/miss", separation)
    # High separation == exploitable membership oracle; partition caches.

serving_isolation_policy:
  caching:
    kv_cache_scope: per_tenant
    cross_tenant_prefix_sharing: false_for_sensitive
  scheduling:
    sensitive_tenants: isolated_batches
    ordering_leak_mitigation: true
  timing:
    pad_to_quantum: sensitive_requests
    speculative_decoding: off_for_sensitive
  metering:
    per_tenant_query_quota: enforced
    leakage_budget_bits: defined

Glossary

Side channel

An unintended information leak arising when secret-dependent behavior becomes observable, such as through timing.

Co-tenancy

Multiple tenants sharing the same serving hardware, so their requests contend for shared resources.

Prompt-prefix caching

Reusing cached key-value computation for a shared prompt prefix, making repeated prefixes faster to process.

Membership oracle

A channel that reveals whether a specific input was recently processed, here via cache-hit timing.

Dynamic batching

Grouping concurrent requests to use an accelerator efficiently, coupling their timing and ordering.

Speculative decoding

Speeding generation by drafting tokens with a small model and verifying them in parallel, with a content-dependent speedup.

Mutual information

A measure of how many bits an observable reveals about a secret, quantifying a side channel's per-query leak.

Cache partitioning

Isolating caches per tenant so a hit can only reflect that tenant's own prior requests.

References

1. Shumailov et al., Sponge Examples: Energy-Latency Attacks on Neural Networks (arXiv 2006.03463)
2. Leviathan et al., Fast Inference from Transformers via Speculative Decoding (arXiv 2211.17192)
3. Kwon et al., Efficient Memory Management for LLM Serving with PagedAttention (arXiv 2309.06180)
4. Kocher, Timing Attacks on Implementations of Diffie-Hellman, RSA, DSS, and Other Systems (CRYPTO 1996)
5. NIST AI 100-2 e2023, Adversarial Machine Learning: A Taxonomy and Terminology