The Arithmetic of One Token — Transformer Mechanics

Ask how expensive a language model is and you get vague answers about size. But the cost of generating one token is not vague at all — it's countable, and the count fits on the back of an envelope. A dense forward pass costs about two floating-point operations per parameter. Attention adds a term that grows with the square of the context. And, surprisingly, when a model is decoding one token at a time, the bottleneck is usually not arithmetic but memory bandwidth. This article derives all three facts from the shapes we established earlier, so you can predict cost instead of guessing it.

Two FLOPs per parameter per token

Almost all of a transformer's compute is matrix multiplication. Consider a single linear layer that maps an input of size d_in to an output of size d_out with a weight matrix W. Producing the output for one token is a matrix-vector product: each of the d_out outputs is a dot product over d_in inputs, costing one multiply and one add per element — two FLOPs. So the layer costs 2 x d_in x d_out FLOPs for one token, which is exactly twice the number of weights in W.

That relationship is completely general: every weight is touched by one multiply and one accumulate per token. Sum it over all the weight matrices in the network and you get the famous estimate — a dense forward pass costs approximately 2P floating-point operations per token, where P is the parameter count. A 7-billion-parameter model spends roughly 14 billion FLOPs per token, before counting attention's own score arithmetic.

📌

The core identity. A matrix-vector product costs two FLOPs per weight, so a dense forward pass is about 2P FLOPs per token.

Where the parameters actually live

To see what dominates P, count the weights in one layer in terms of the model dimension d. The attention block has four d-by-d projection matrices — W_Q, W_K, W_V, and W_O — for roughly 4d^2 parameters. The feed-forward MLP typically expands to a hidden width of 4d and back, giving two matrices of size d-by-4d, or about 8d^2 parameters. So the MLP holds roughly twice the weights of attention, and per-layer parameters are on the order of 12d^2.

This is why, at short context lengths, the 2P term is dominated by the MLP, not attention's projections. It also means most of a model's stored knowledge and most of its FLOPs live in those feed-forward matrices — a useful thing to remember when reasoning about what a model can hold.

The feed-forward MLP holds about twice the weights of the attention projections.

The attention term: order n-squared d

The 2P count covers the projections but not the attention interaction itself, which has its own arithmetic. Forming the score matrix QK^T is an n-by-d_k times d_k-by-n multiply per head, costing on the order of n^2 x d_k; summed over heads that's about n^2 x d. Applying the softmax weights to V costs another n^2 x d. So the attention interaction is order n^2 x d per layer — quadratic in sequence length.

Compare the two terms. The dense projections cost order n x d^2 to process n tokens (that's the 2P term times n). The attention interaction costs order n^2 x d. Their ratio is n over d: attention overtakes the projections once the sequence length n approaches the model dimension d. For a model with d around 4096, contexts well beyond a few thousand tokens push attention into the dominant cost — the reason long-context inference feels so different.

Per token, cost splits into dense projections, the quadratic attention term, and growing KV-cache bytes.

Prefill versus decode

These counts play out differently in the two phases. Prefill processes all n prompt tokens at once, so its matmuls are matrix-matrix products with large inner dimensions — the hardware runs at high utilization, and prefill is compute-bound. Decode generates one token per step, turning those same matmuls into matrix-vector products with a batch of one, which the hardware finds far harder to keep busy.

The FLOP counts per token are similar in both phases (thanks to the KV cache, decode still only does about 2P of dense work per token). What differs is efficiency: prefill saturates the arithmetic units; decode leaves them idle waiting on memory. To see why, we have to count bytes moved, not just operations.

Why decode is memory-bound

During single-token decode, producing one token requires reading every weight in the model out of memory exactly once to multiply it against the one input vector. That's about P weights, or on the order of 2P bytes in half precision, streamed from memory per token. But the arithmetic performed on each weight is just a couple of FLOPs. The ratio of compute to memory traffic — the arithmetic intensity — is therefore tiny, roughly one operation per byte.

Modern accelerators can do far more arithmetic per byte of memory bandwidth than that, so decode leaves the compute units starved and idle: the token rate is set by how fast weights can be read from memory, not by FLOPs. This is the single most important performance fact about autoregressive inference, and it's why techniques like batching, quantization, and speculative decoding — all covered in the inference-optimization articles — target memory traffic rather than raw compute.

Prefill saturates the arithmetic units; decode is throttled by memory bandwidth.

Counting the KV-cache bytes

The other quantity worth counting is the memory the KV cache consumes, because it, not FLOPs, often sets the practical context limit. Each token stores a key and a value vector in every layer. So the per-token cache size is roughly 2 (key and value) times the number of layers times the number of key/value heads times the head dimension times the bytes per number. Multiply by the sequence length to get the total.

For a mid-sized model this can be tens to hundreds of kilobytes per token, so a long conversation of thousands of tokens runs into gigabytes of cache. That linear-in-length growth is exactly why grouped-query attention and cache compression exist, and why context windows are capped where they are — the cache, not the compute, hits the wall first.

Why this matters

With these three counts — 2P FLOPs of dense work, an n^2 d attention term, and a KV cache linear in length — you can estimate the cost and the bottleneck of almost any inference scenario without running it. Short prompt, short answer: dense-bound and cheap. Huge context: attention and cache dominate. Streaming one token at a time: memory-bandwidth-bound regardless of how many FLOPs the chip could do.

The next article steps back to two structural pieces we've leaned on but not examined: how positional information enters a permutation-blind attention mechanism, and how the residual stream carries and combines everything each layer writes.

Key takeaways

A matrix-vector product costs two FLOPs per weight, so a dense forward pass is about 2P FLOPs per token.
Per layer, attention projections hold about 4d^2 parameters and the MLP about 8d^2, so the MLP dominates P.
The attention interaction costs order n^2 d per layer and overtakes the dense term once n approaches d.
Prefill is compute-bound (matrix-matrix); single-token decode is memory-bound (matrix-vector).
Decode must stream ~2P bytes of weights per token but does only ~2 FLOPs each, giving tiny arithmetic intensity.
The KV cache grows linearly with context and often sets the practical limit before compute does.

Glossary

FLOP: One floating-point operation; a multiply and an add count as two.
2P estimate: The rule that a dense forward pass costs about two FLOPs per parameter per token.
Arithmetic intensity: The ratio of compute performed to bytes moved from memory.
Compute-bound: A regime where speed is limited by arithmetic throughput, as in prefill.
Memory-bound: A regime where speed is limited by memory bandwidth, as in single-token decode.
n^2 d term: The order of attention's interaction cost per layer, quadratic in sequence length.
KV-cache size: Roughly 2 x layers x kv-heads x head-dim x bytes per token, growing linearly with length.