Why Long Chats Get Slower and Pricier — Inside a Language Model

You may have noticed that a quick question gets a quick answer, but pasting in a long document makes the model pause before it responds — and if you pay to use one, longer conversations cost more. None of that is arbitrary. It falls straight out of the ideas we've built up: words as numbers, attention comparing every word with every other, one-word-at-a-time writing, and the memory cache. This final article ties them together to explain, in plain terms, why length drives both speed and cost.

Two kinds of work a model does

Every request splits into two jobs. First the model has to read your input — the prompt, the conversation so far, any document you pasted. Then it has to write its reply, one word at a time, the way we saw earlier. These two jobs have very different cost shapes, and telling them apart is the key to the whole picture.

Reading the input is often called prefill: the model processes everything you gave it and fills up its memory cache in one big burst. Writing the reply is called decode: the slow, steady, word-by-word loop. Your total wait and your total cost are just these two added together.

The model first reads all your input in one burst, then writes the reply one word at a time.

Reading grows faster than you'd expect

Here's the part that surprises people. When the model reads your input, attention has every word look at every other word. With ten words that's a manageable number of comparisons. But double the input to twenty words and the number of word-to-word comparisons doesn't just double — it roughly quadruples, because each of the extra words has to be compared against all the others too.

This 'every word looks at every word' pattern means the reading work grows with the square of the length: ten times more words means roughly a hundred times more comparisons. That explosive growth is exactly why feeding in a very long document causes a real pause before the first word of the reply appears.

Because every word compares with every other, ten times the words means roughly a hundred times the work.

Writing costs a steady amount per word

The writing job has a gentler shape. As we saw, producing each output word is one full run of the model, and that run costs about the same each time — a fixed chunk of arithmetic set by the model's size. A bigger model (more of those internal numbers, called parameters) costs more per word; a smaller one costs less. But for a given model, word number five hundred costs roughly the same to produce as word number five.

So the writing bill is simple: it's basically the fixed per-word cost multiplied by how many words the model writes. Ask for a long essay and you pay for every word of it, at a steady rate.

📌

A handy rule. Reading cost grows with the square of the input length; writing cost grows straight in line with how many words you ask for.

How the cache changes the picture

The memory cache from the last article softens the worst of this. Without it, every single output word would re-read the whole conversation, and that square-shaped cost would strike on every word — catastrophic for long chats. The cache turns that into a quick scan of saved keys and values, so each new word stays cheap even far into the conversation.

But the cache doesn't make length free — it just moves the pressure. Now the limit is memory: the cache grows with every word, so a very long conversation needs a lot of fast memory to hold it. That's why models publish a maximum context window, the largest amount of text they'll hold at once. Fill it up and there's simply no room for more.

So what actually drives your bill

Put it together and the cost of any interaction comes down to a short list. Longer inputs cost more to read, and that reading cost climbs steeply. Longer outputs cost more to write, at a steady per-word rate. And the whole conversation history usually rides along in every new request, so a long-running chat quietly carries all its past text into each new turn.

This is why the same question can cost pennies or pounds depending on context. A one-line prompt with a short answer is cheap. The same prompt buried under a huge pasted document, asking for a long response, in a conversation that's already ten messages deep, is a different animal entirely.

Three things set the bill: how much you send, how much the model writes, and how much history rides along.

Practical ways to keep it lean

Understanding the cost shape hands you some easy wins. Because reading grows steeply with length, trimming a bloated prompt or summarising an old conversation instead of resending it in full can make a real difference. Because writing is charged per word, asking for a concise answer rather than an exhaustive one directly lowers the cost.

None of this requires touching the model — it's all about what you feed it and what you ask for. The same instincts that make writing clear also make it cheap: say what you need, give only the context that matters, and ask for the length you actually want.

Trim or summarise long context instead of pasting everything every time.
Ask for the answer length you actually need — extra words cost extra.
Start a fresh conversation when the old history is no longer relevant.
Remember the context window is finite: very long threads eventually hit a wall.

Where this leaves you

That completes the journey. You've followed a message from raw text all the way through: words become tokens, tokens become embeddings, attention lets each word gather context, the model writes one word at a time, a cache lets it remember its own work, and the shapes of reading and writing explain why length drives speed and cost. No single step is magic — each is a small, sensible idea, and together they produce something remarkable.

If you'd like to go one level deeper — to see the actual shapes, the query-key-value maths, and how engineers squeeze more speed out of every step — the advanced companion series, 'Inside the Transformer', picks up exactly where this one ends.

✅

You've got the whole picture. Length ties speed and cost together because reading grows steeply and writing is charged per word — everything else follows from that.

Key takeaways

Every request has two jobs: reading the input (prefill) and writing the reply word by word (decode).
Reading cost grows with the square of the input length, because every word compares with every other word.
Writing cost is steady per word, set by the model's size, and adds up with how many words you request.
The cache keeps per-word writing cheap but shifts the limit to memory, which is why context windows are capped.
Your bill is driven by input length, output length, and the chat history carried into each turn.
You can cut cost by trimming context, asking for concise answers, and starting fresh when history is stale.

Glossary

Prefill: The stage where the model reads all your input at once and fills its cache.
Decode: The word-by-word stage where the model writes its reply.
Parameters: The model's internal numbers; more of them means a bigger, costlier-per-word model.
Context window: The largest amount of text a model will hold at once, measured in tokens.
Quadratic growth: Growth with the square of the size — double the length, roughly four times the work.
Token: A chunk of text, and the unit in which length and cost are usually measured.