The Memory Trick: How Models Remember the Conversation

In the last article we found a puzzle hiding inside the writing loop. To produce each new word, a model reads everything that came before it — and it does that afresh for every single word. In a long conversation that sounds like it should get slower and slower, re-reading a growing pile of text thousands of times. It would, except for one elegant shortcut that almost every modern model uses. It's called the KV cache, and it's the reason long chats stay quick. This article explains it with no maths at all.

The problem: re-reading everything, every word

Recall how generation works: the model reads all the text so far, predicts one word, adds it, and repeats. Now think about what 'reads all the text so far' really costs. For the tenth word, the model processes ten words. For the hundredth word, it processes a hundred. For the thousandth, a thousand. The work per word keeps climbing as the conversation grows.

And most of that work is repeated. When the model got to word ten, it already did all the thinking about words one through nine. Then at word eleven it throws that away and thinks about words one through ten again — redoing almost exactly the same effort. That repetition is the waste we want to eliminate.

⚠️

The wasteful part. Every new word re-examines all the earlier words from scratch, redoing work that was already done moments ago.

A quick reminder of what attention produces

To see the fix, remember the three roles from the attention article. Every word produces a key (a label advertising what it's about) and a value (its actual content), plus a query when it's the one doing the looking. When a new word arrives, it forms its query and compares it against the keys of all the earlier words, then blends in their values.

Here's the crucial observation: an earlier word's key and value never change. The key and value for the third word are the same whether we're generating word four, word forty, or word four hundred. So recomputing them every time is pure waste — they were already worked out the first time that word appeared.

The fix: remember instead of recompute

The trick is almost embarrassingly simple: the first time the model works out a word's key and value, it saves them in a little memory store and never computes them again. That store is the KV cache — 'KV' for keys and values, 'cache' for a stash of things kept handy for reuse. From then on, when a new word needs to look back, it just reads the saved keys and values straight out of the cache.

So the work per new word collapses. Instead of reprocessing the entire conversation, the model only has to do the fresh work for the one brand-new word, then glance at a cupboard of already-computed keys and values for everything before it.

Without the cache the model redoes every earlier word; with it, only the new word needs fresh work.

What actually gets stored

You can picture the KV cache as a growing list, one slot per word the model has seen. Each slot holds two things: that word's key and that word's value. When word one is processed, its key and value go into slot one. Word two fills slot two. And so on — the cache just keeps growing by exactly one slot for each new word, and old slots are never touched again.

When the model generates the next word, it produces a query for that new word, checks it against every key already sitting in the cache, and pulls in the matching values. Then it adds its own key and value as a fresh slot, ready for the word after it. The cache is simply the model's running memory of the conversation.

Each word adds one slot holding its key and value; earlier slots are reused, never recomputed.

Why this makes long chats fast

With the cache in place, generating word one thousand isn't dramatically more expensive per step than generating word ten. The new-word work is roughly constant, and the only growing part is a quick scan over the saved keys and values — cheap compared with recomputing them all. That's why a model can keep replying at a steady, snappy pace even deep into a long conversation.

This is also why the very first response to a long document can take a noticeable moment: the model has to fill the cache for all that input up front, a stage often called 'prefill'. After that initial fill, each new word rides on the cache and comes out quickly.

✅

The payoff. By reusing saved keys and values, the model does roughly constant fresh work per word instead of re-reading the whole conversation.

The catch: the cache takes up memory

Nothing is free. The KV cache lives in the computer's fast memory, and it grows with every word — a longer conversation means a bigger cache. Double the length of the chat and you roughly double the amount of memory the cache needs. For very long conversations or documents, this memory can become the real limit, more than raw speed.

That trade — save time by spending memory — is the hidden economics behind every long chat. Exactly how that memory and the growing scan add up to real-world speed and cost is the subject of the final article in this series.

Why this matters

The KV cache is one of those unglamorous engineering ideas that quietly makes the whole experience possible. Without it, long conversations would crawl and cost far more. With it, a model can carry a lengthy thread and still answer at a comfortable pace, because it remembers its own past work instead of grinding through it again.

It also explains behaviours you may have noticed: the little pause before a reply to a long prompt, the steady typing speed thereafter, and the fact that enormous conversations can bump into limits. Keep this picture of a growing memory of keys and values in mind — it's the bridge to understanding why length, speed, and cost are all tied together, which is where we finish next.

Key takeaways

Generating each word means re-reading all earlier words, which repeats work that was already done.
An earlier word's key and value never change, so recomputing them every step is pure waste.
The KV cache saves each word's key and value once and reuses them, hence 'keys and values cache'.
The cache grows by one slot per word; new words look up the saved slots instead of recomputing them.
This keeps the fresh work per word roughly constant, so long chats stay fast.
The trade-off is memory: the cache grows with conversation length and can become the real limit.

Glossary

KV cache: A store of every past word's key and value, kept so they can be reused instead of recomputed.
Key: A label a word produces advertising what it is about, used for matching during attention.
Value: A word's actual content, pulled in when it is attended to.
Cache: A stash of already-computed results kept handy so they don't have to be worked out again.
Prefill: The initial stage where the model fills the cache for all the input text before generating.
Memory: The computer's fast working storage, where the KV cache lives and grows with the conversation.