When a chatbot streams out a fluent paragraph, it's tempting to imagine it composing the whole thought and then typing it out. That's not what happens. Underneath, the model is doing something far simpler and stranger: it predicts just one next word, writes it down, and then reads everything so far to predict the word after that. A polished essay is nothing more than that tiny guess, repeated hundreds of times. This article follows that loop step by step.

The one job a model actually has

Strip away the magic and a language model does exactly one thing: given some text, it guesses what word is most likely to come next. That's it. You've used a smaller version of this every time your phone suggests the next word above the keyboard. Type 'Happy new' and it offers 'year'. A language model is that same idea, scaled up enormously and trained on a huge amount of writing.

In the previous articles we saw how words become embeddings and how attention lets each word gather context. All of that machinery exists to serve this one final step: turn everything read so far into a single, well-informed guess about the next word.

📌
The whole trick. A language model only ever answers one question: given the text so far, what is the most likely next word?

The loop: predict, add, repeat

One guess doesn't make a sentence, so the model wraps that guess in a loop. It reads everything written so far, predicts the next word, and adds that word to the end of the text. Then it reads the whole thing again — now one word longer — and predicts the next word after that. Round and round it goes, each pass adding exactly one word.

Because every new word is fed back in before the next guess, the model always writes with full awareness of what it has already said. This step-by-step, feed-it-back style has a name: it's called autoregressive generation. The word 'autoregressive' just means 'using its own previous output as the next input'.

The model reads what's written, predicts one word, adds it, then reads again — over and over. The writing loop repeat Read so far all the text yet Predict next word one best guess Add it append to the text
The model reads what's written, predicts one word, adds it, then reads again — over and over.

How it picks: a probability for every word

When the model predicts the next word, it doesn't blurt out a single answer. It produces a probability for every word in its vocabulary at once — a giant scorecard saying 'mat: 55% likely, floor: 20%, sofa: 15%, roof: 10%', and so on for tens of thousands of options. Most words get almost zero; a handful get most of the probability.

Picture the sentence 'The cat sat on the ___'. The model's scorecard might look like the bars below: 'mat' way out in front, a few plausible runners-up, and everything else vanishingly unlikely. Choosing the next word means picking from this ranked list.

For 'The cat sat on the ___', the model scores every possible next word; a few lead the pack. Next-word odds mat 55% floor 20% sofa 15% roof 10%
For 'The cat sat on the ___', the model scores every possible next word; a few lead the pack.

Why it doesn't always pick the top word

If the model always grabbed the single highest-scoring word, its writing would be correct but robotic and repetitive — and it would give the exact same answer every time. So instead it usually samples: it picks from the top candidates with a bit of chance mixed in, favouring likely words but occasionally taking a less obvious one. That controlled randomness is what makes the output feel natural and varied.

There's a dial for this, often called temperature. Turn it low and the model plays it safe, almost always choosing the front-runner — great for facts and code. Turn it up and the model gets more adventurous, reaching for surprising words — handy for brainstorming or creative writing. It's the same model throughout; only how boldly it picks changes.

Low temperature keeps output safe and predictable; high temperature makes it more varied and creative. The temperature dial Low safe, repetitive Medium balanced High creative, risky
Low temperature keeps output safe and predictable; high temperature makes it more varied and creative.

Where the writing stops

If the model just kept looping, it would never shut up. So its vocabulary includes a special invisible token that means 'I'm done'. During training the model learns to predict this stop token when a thought is complete — after a full answer, the end of a sentence, or a natural finish. When that token comes up, the loop ends and you see the reply as finished.

There's usually also a hard limit — a maximum number of words the model is allowed to generate in one go — as a safety net. Whichever comes first, the stop token or the limit, brings the loop to a halt.

Every new word is a full pass through the model

Here's the detail that shapes everything about speed and cost. Producing each single word means running the entire model once — all its layers, all its attention, all its billions of internal numbers — just to add one word. Then the model does it all again for the next word. A hundred-word answer is roughly a hundred full runs of the model, one after another.

That's why replies appear to 'type' themselves out rather than popping up instantly, and why a longer answer takes proportionally longer to produce. It's also the root of why these systems cost what they do to run — a thread we'll pull on in the last two articles of this series.

⚠️
The hidden cost. Each word you see is one complete run of the model, so longer outputs mean proportionally more work.

Why this matters

Once you see that a model writes one word at a time, a lot of its behaviour stops being mysterious. It streams because it genuinely produces text incrementally. It can lose track or contradict itself late in a long answer because each word is chosen in the moment, not planned in advance. And it can be steered — via temperature and the words you've already given it — because every guess leans on everything that came before.

But there's a problem lurking in that loop. If the model re-reads the entire text on every single word, doesn't the work pile up as the conversation grows? It does — and the clever fix for it, the memory trick that keeps long chats affordable, is exactly where the next article picks up.

Key takeaways

  • A language model's only job is to predict the single most likely next word given the text so far.
  • It generates by looping: read everything, predict one word, add it, then read again — called autoregressive generation.
  • Each prediction is a probability score across the model's entire vocabulary, with a few words leading.
  • The model samples rather than always taking the top word; a temperature dial controls how bold those picks are.
  • A special stop token (or a length limit) ends the loop when the answer is complete.
  • Producing each word is one full run of the model, so longer answers take proportionally more time and cost.

Glossary

Next-word prediction
The model's core task: guessing the most likely word to follow the text so far.
Autoregressive generation
Writing by feeding the model's own output back in to predict the next word, one at a time.
Vocabulary
The full set of words and word-pieces a model can choose from.
Probability
A score for how likely each possible next word is.
Sampling
Choosing the next word with a bit of chance rather than always taking the top pick.
Temperature
A dial that controls how safe or adventurous the model's word choices are.
Stop token
A special signal the model predicts when it has finished its answer.