The Forward Pass, End to End

A transformer can look like an intimidating wall of matrices, but its top-level structure is remarkably regular: embed the input, push it through a stack of identical blocks, then project back out to vocabulary space. Every impressive behaviour these models show is one traversal of that pipeline — a forward pass. This article maps the whole route at a systems level, naming each stage and the tensor shapes flowing between them, so the deeper articles on attention mechanics have a scaffold to hang on. It assumes you're comfortable with vectors and matrix multiplication, but nothing about transformers themselves.

The shape of the whole computation

Fix a decoder-only model — the family behind modern generative language models. Its forward pass takes a sequence of token IDs and returns, for the final position, a probability distribution over the vocabulary: the model's guess for the next token. Between those two ends sits a fixed pipeline that never changes shape, only depth.

Concretely: the token IDs index into an embedding table to become vectors of width d (the model dimension). Those vectors flow through L identical transformer blocks, each refining them while preserving their shape. A final normalization and a projection to vocabulary size V produce logits, and a softmax turns those into probabilities. The representation width d is conserved end to end — every block reads a d-wide vector per token and writes a d-wide vector back.

The residual stream: the model's working memory

It helps to think of the d-wide vector at each token position as a residual stream — a running accumulator that every block reads from and adds to. A block never overwrites this vector wholesale; it computes an update and adds it back. That additive discipline is what the 'residual connections' in the architecture provide, and it's the reason gradients flow cleanly through very deep stacks during training.

Under this view, each layer's job is to write useful information into the residual stream: attention writes contextual mixing, the feed-forward sublayer writes learned transformations. Later layers read whatever earlier layers deposited. We'll return to the residual stream in depth in a dedicated article; for now, hold the picture of one persistent vector per position, growing richer as it ascends the stack.

Inside one block: attention, then MLP

Every transformer block has two sublayers in sequence. First comes multi-head self-attention, which lets each position mix in information from other positions. Second comes a position-wise feed-forward network (a two-layer MLP), which transforms each position independently. Each sublayer is wrapped identically: normalize the input, apply the sublayer, then add the result back to the residual stream — the pattern often written as x + Sublayer(Norm(x)).

Modern models overwhelmingly use pre-normalization: the layer norm sits before the sublayer, not after, which stabilizes training at depth. The two sublayers play complementary roles — attention moves information across positions, the MLP does the heavy per-position computation and holds much of the model's stored knowledge. Stacking L of these blocks is the entire body of the network.

From the last block to a distribution

After the final block, the residual stream passes through one last normalization, then a linear projection into vocabulary space — the unembedding, a matrix of shape d by V. This yields a vector of V logits per position: one raw score for every token the model knows. In many models the unembedding weights are tied to the input embedding table, reusing the same parameters.

For generation we only care about the last position's logits. A softmax converts them into a probability distribution, and a sampling step (greedy, temperature, top-p, and so on) picks the next token. That token is appended and the whole forward pass runs again — the autoregressive loop. Everything upstream exists to make that final row of logits as informed as possible.

Why it's called a forward pass

The term contrasts with the backward pass of training. During training, a forward pass computes predictions and a loss, then backpropagation runs the computation in reverse to get gradients and update the weights. At inference time there is no backward pass: weights are frozen, and each generated token is produced by a single forward traversal of the frozen network.

This distinction matters for reasoning about cost. Inference latency and FLOPs are governed entirely by the forward pass, executed once per generated token (plus one bulk pass over the prompt). The next article quantifies that per-token arithmetic; the rest of the series dissects the attention sublayer that sits at the heart of every block.

What this series covers — and what it doesn't

This series is about mechanics: the exact shapes and operations inside the forward pass, especially attention. Across the coming articles we'll open up multi-head attention, count the arithmetic of a single token, examine positional information and the residual stream, and trace how the KV cache falls naturally out of the attention equations.

It deliberately stops at the boundary of serving-level optimization. Techniques for making inference faster or cheaper at scale — speculative decoding, continuous batching, paged attention, quantization — are covered in separate advanced articles on inference optimization and speculative decoding. Here we care about what the model computes, not how a serving stack schedules it.

Why this matters

Holding the end-to-end pipeline in your head turns the rest of the details into a set of well-placed pieces rather than a fog of matrices. When we later derive attention's cost or explain why the KV cache exists, you'll be able to locate exactly where in the forward pass that cost lands and which tensor shapes drive it.

With the map in place, we descend into the single most important sublayer. The next article opens up multi-head self-attention: the Q, K, V projections, the scaled dot-product, the head split, and the shapes that make it all line up.

Glossary

Forward pass

One traversal of the network from inputs to outputs, used to produce predictions at inference.

Decoder-only transformer

The architecture behind generative language models, using causal self-attention over past tokens.

Model dimension (d)

The width of the per-token representation, preserved through every block.

Residual stream

The persistent per-token vector that each block reads from and additively updates.

Sublayer

One of a block's two components — self-attention or the feed-forward MLP.

Pre-normalization

Placing layer norm before each sublayer, which stabilizes training of deep stacks.

Unembedding

The final linear projection from the model dimension to vocabulary size, producing logits.

Logits

Raw per-token scores over the vocabulary, before softmax turns them into probabilities.