Two pieces of transformer machinery are easy to take for granted because they sit in the background of the attention equations: where word order comes from, and how information moves between layers. Neither is optional. Self-attention on its own cannot tell 'dog bites man' from 'man bites dog', and without a disciplined communication channel a hundred-layer stack could not combine what its layers learn. This article examines both — positional encoding and the residual stream — at the level of what they compute and why they take the forms they do.

Attention is permutation-blind

Look back at the scaled dot-product: the output for position i is a weighted sum over the values of all positions, with weights from query-key dot products. Nothing in that expression references where a token sits in the sequence. Permute the input tokens and, apart from relabeling, the same set of outputs comes back. Formally, self-attention is permutation-equivariant — it treats its input as a set, not a sequence.

That is a fatal gap for language, where order carries meaning. So order has to be added deliberately, as an extra signal the model can read. Every transformer therefore includes a positional encoding scheme whose only job is to break this symmetry and tell attention which token came where.

⚠️
The structural gap. Self-attention treats its input as an unordered set, so positional information must be injected explicitly.

Absolute positional encodings

The original approach adds a position-dependent vector to each token embedding before the first block. In the classic sinusoidal scheme, each dimension of the position vector is a sine or cosine of the position at a different frequency, giving every position a unique, smoothly varying fingerprint. A simpler variant just learns a separate embedding per position index. Either way the block input becomes token meaning plus a marker of where it sits.

Absolute encodings are easy to reason about but have a weakness: the model sees position 5 and position 105 as unrelated points, and it cannot naturally generalize to sequence lengths longer than it trained on. What usually matters for language is not absolute position but the distance between tokens — which motivates a different design.

A positional signal is combined with each token embedding to form the block input. Injecting position Token embedding what the word is Positional signal where it sits Block input meaning + order
A positional signal is combined with each token embedding to form the block input.

Rotary position embeddings

Modern models overwhelmingly use rotary position embeddings (RoPE). Instead of adding a position vector, RoPE rotates the query and key vectors by an angle proportional to their position, applying the rotation in two-dimensional slices of the head. The elegant consequence: when a rotated query at position i is dotted with a rotated key at position j, the result depends only on the relative offset i minus j, not on either absolute position.

This bakes relative-position awareness directly into the attention scores at no extra parameters, and it extrapolates far more gracefully to longer contexts — which is why RoPE, often with frequency adjustments for context extension, underpins most current long-context models. The takeaway is the shift in what the geometry encodes: from 'where you are' to 'how far apart we are'.

Absolute schemes add a position vector; rotary encoding rotates Q and K to capture relative distance. Two ways to encode order Absolute add a position vector Rotary (RoPE) rotate Q and K vs
Absolute schemes add a position vector; rotary encoding rotates Q and K to capture relative distance.

The residual stream as a communication channel

Recall that each block adds its output back to a persistent per-token vector rather than replacing it. Viewed across the whole stack, that vector is a shared bus — the residual stream — that every sublayer reads from and writes to. Attention heads and MLPs read the current state, compute an update, and add it in. Because the operations are additive, later layers can read what earlier layers deposited, and the model can route information across depth by writing it into the stream and picking it up further along.

A powerful way to think about it: the residual stream is high-dimensional bandwidth that components communicate through. Different heads tend to write into different subspaces, and the model can pack more features than dimensions by storing them in superposition — near-orthogonal directions that rarely interfere. This channel view is the foundation of modern interpretability work on transformers.

Each sublayer reads the residual stream, computes an update, and adds it back for later layers to read. The residual bus add add Residual stream shared per-token bus Attention writes cross-position info MLP writes per-position features Later layers read accumulated state
Each sublayer reads the residual stream, computes an update, and adds it back for later layers to read.

Normalization keeps the stream in range

Adding update after update into the residual stream could let its scale drift, destabilizing training. Layer normalization counters this by rescaling each sublayer's input to a controlled statistical range. Modern models place it before each sublayer — pre-normalization — which keeps a clean, unnormalized residual path running straight through the network and makes very deep stacks trainable.

Many current models use RMSNorm, a cheaper variant that normalizes by the root-mean-square of the vector without centering it. The detail to carry forward is the pattern: normalize a copy of the stream, feed it to the sublayer, add the sublayer's output back to the untouched stream. The residual path itself is never overwritten, only added to.

Why this matters

These two ideas explain properties you rely on daily. Positional encoding — especially RoPE — is why a model tracks word order and why context-length extension is even possible. The residual stream is why gradient flow survives great depth, why interpretability researchers can trace features across layers, and why the 'add, don't overwrite' discipline appears everywhere in the architecture.

We now have every mechanical piece: the forward-pass map, multi-head attention, the per-token arithmetic, and the positional and residual scaffolding. The final article brings them together to show how the KV cache falls directly out of the attention equations under causal masking — closing the loop between mechanics and the inference behaviour we started from.

Key takeaways

  • Self-attention is permutation-equivariant, so positional information must be added explicitly.
  • Absolute encodings add a per-position vector (sinusoidal or learned) but generalize poorly to new lengths.
  • Rotary embeddings (RoPE) rotate Q and K so scores depend on relative offset, extrapolating better to long contexts.
  • The residual stream is a shared additive bus every sublayer reads from and writes to across depth.
  • Components write features into near-orthogonal subspaces, enabling superposition and cross-layer communication.
  • Pre-normalization (often RMSNorm) keeps the stream in range while leaving the residual path unoverwritten.

Glossary

Permutation-equivariant
Reordering the inputs reorders the outputs identically — attention treats input as a set.
Positional encoding
A scheme that injects token-order information into the model.
Sinusoidal encoding
Absolute positions represented by sines and cosines at varying frequencies.
Rotary position embedding (RoPE)
Rotating Q and K by a position-dependent angle so scores depend on relative distance.
Residual stream
The shared per-token vector that all sublayers additively read from and write to.
Superposition
Packing more features than dimensions by storing them in near-orthogonal directions.
Pre-normalization
Normalizing a sublayer's input while leaving the residual path unnormalized.
RMSNorm
A normalization that scales by root-mean-square without subtracting the mean.