Research

Watch · narrated whiteboard episodesL3

Inference-Infrastructure Side Channels: Leaks in Shared Serving

Multi-tenant, batched, KV-cached, speculatively-decoded serving is efficient — and leaky. Timing and cache behavior turn a shared endpoint into a side channel. This threat lab formalizes the shared-serving threat model, prompt-cache membership oracles, decoding-timing leaks measured in bits per query, denial-of-wallet resource amplification, and the isolation trade-offs that mitigate them — each paired with a hardening. Grounded in the primary systems and side-channel literature.

Murali Chillakuru·5 episodes
  1. 14 min Episode 1The Shared-Serving Threat Model: Co-Tenancy, Batching, and the Attacker's ObservablesA moderator and an expert map out why the optimizations that make model serving cheap — shared caches, dynamic batching, speculative decoding — give a co-tenant a measurable window into other users' traffic.
  2. 12 min Episode 2KV-Cache and Prompt-Cache Leakage: Prefix Caching as a Membership OracleA moderator and an expert examine how prefix caching, designed to make repeated system prompts cheaper, creates a timing oracle that reveals whether another user recently submitted a given prompt.
  3. 13 min Episode 3Timing Side Channels in Decoding: Speculative Acceptance and Bits-per-QueryA moderator and an expert examine how speculative decoding — a performance optimization that speeds up generation for predictable text — inadvertently encodes content predictability in response timing.
  4. 13 min Episode 4Denial and Resource Amplification: Sponge Inputs and Denial-of-WalletA moderator and an expert examine how input-dependent computation cost gives attackers a lever to crash availability or drain budgets — without ever needing to break authentication.
  5. 16 min Episode 5Isolation and Mitigations: Cache Partitioning, Quotas, and the Throughput CostA moderator and an expert assemble the full defense program for serving-infrastructure side channels — showing that every fix costs throughput and the real work is choosing those trades deliberately.