Read this sentence: 'The trophy didn't fit in the suitcase because it was too big.' What does 'it' mean — the trophy or the suitcase? You know instantly it's the trophy, because a big trophy is the reason something wouldn't fit. To answer, your mind quietly connected 'it' to the right earlier word. A language model faces this exact challenge on every word it reads, and the mechanism it uses to solve it is called attention. It is the single most important idea inside these models, and this article builds it up from scratch.

The problem: words depend on other words

In the last article we saw that a model turns each word into an embedding — a list of numbers capturing its meaning. But there's a catch: a word's real meaning often depends on the words around it. The word 'bank' means something different in 'river bank' versus 'savings bank'. The word 'it' means nothing on its own; it borrows its meaning from an earlier word. So a fixed fingerprint for each word, all by itself, isn't enough.

What the model needs is a way for each word to look at the other words in the sentence and pull in whatever context it needs. That looking-around is exactly what attention does.

📌
The core problem. A word's meaning depends on its neighbours, so each word needs a way to gather relevant context from the rest of the sentence.

The big idea: let every word look around

Attention works like this: for every word, the model lets that word scan all the other words and decide how much to 'pay attention' to each one. Words that are highly relevant get a lot of attention; words that aren't get very little. Then the word updates its own understanding by blending in a little of the words it cared about most.

Take our earlier puzzle. When the model processes the word 'it', attention lets 'it' look back at 'trophy', 'suitcase', and the rest — and lean most heavily on 'trophy'. After attention, the fingerprint for 'it' quietly carries a bit of 'trophy' inside it. That's how the model 'knows' what 'it' refers to.

When the model processes 'it', attention lets it look at the other words and lean most on the relevant one. Looking around trophy very relevant suitcase some relevance big a little the word 'it' needs context
When the model processes 'it', attention lets it look at the other words and lean most on the relevant one.

Three questions behind every look

How does a word decide who to pay attention to? The model gives every word three roles, and the easiest way to picture them is a library search. First, the word forms a query — the thing it's looking for, like typing a question into the library catalogue. Second, every other word offers a key — a short label advertising what it's about, like the title on a book's spine. Third, every word also holds a value — its actual content, like the text inside the book.

So attention is really a search: the query from one word is compared against the key of every other word to find the best matches. The words whose keys match the query best are the ones worth reading — and their values are what gets pulled in. These three ingredients, query, key, and value, are the whole vocabulary of attention.

A word's query is matched against every other word's key; the best matches hand over their value. Query, key, value Query what I'm looking for Key what each word offers Value the content pulled in
A word's query is matched against every other word's key; the best matches hand over their value.

Scoring: how much attention to pay

For each pair of words, the model compares a query with a key and gets a match score — a single number saying how well they fit. A high score means 'very relevant, pay attention'; a low score means 'ignore this one'. Every word gets a score against every other word, so the model ends up with a whole grid of relevance scores.

Raw scores are awkward to work with, so the model converts each word's scores into percentages that add up to 100. This step is called softmax, and all you need to know is that it turns a jumble of scores into a tidy set of attention weights — how much of the word's focus goes to each neighbour. For our 'it' example, the weights might land like the bars below: most of the attention on 'trophy', a little on the others.

The scores become percentages that add up to 100 — here, most of the focus lands on trophy. Attention weights trophy 70% suitcase 20% big 10%
The scores become percentages that add up to 100 — here, most of the focus lands on 'trophy'.

Mixing it all together

Once the weights are set, the model does the blending. It takes the value from every word, multiplies each by that word's attention weight, and adds them all up. Words with big weights contribute a lot; words with tiny weights barely register. The result is a fresh, context-aware fingerprint for the original word — the old meaning plus a dose of whatever mattered most around it.

That single operation — score, weight, blend — is one round of attention. The word 'it' walks in as a vague pronoun and walks out carrying the meaning of 'trophy'. Multiply this across every word in the sentence, all at once, and the whole passage sharpens into focus.

Why it's called self-attention

When the words doing the looking and the words being looked at come from the same sentence, we call it self-attention — the sentence attending to itself. That's the version at work inside modern language models. Every word simultaneously plays all three roles: it poses its own query, offers its own key, and holds its own value, all in parallel.

Real models don't do this just once. They run several attention steps side by side, each free to focus on a different kind of relationship — one might track what refers to what, another might follow grammar. Blending many such views is what gives these models their surprising grasp of language. We'll meet that idea again, in more detail, in the advanced companion series.

Why this matters

Attention is the engine room of a modern language model. Almost everything impressive these systems do — following a long instruction, keeping track of who did what, staying on topic across paragraphs — comes down to words attending to the right other words. When you hear the word 'transformer', the 'attention' mechanism is the T-shaped heart of it.

Now that each word carries the right context, the model is ready to do something useful with it: predict what word should come next. That's the subject of the next article, where we watch a model write a sentence one word at a time.

✅
The takeaway. Attention lets every word gather context from the words that matter most, turning isolated fingerprints into meaning that fits the whole sentence.

Key takeaways

  • A word's meaning depends on its neighbours, so each word needs a way to gather context.
  • Attention lets every word look at all the others and decide how much to focus on each.
  • Each word plays three roles: a query (what it seeks), a key (what it offers), and a value (its content).
  • Queries are matched against keys to get scores, which softmax turns into attention weights adding up to 100%.
  • The word's new fingerprint is a weighted blend of the values it paid most attention to.
  • When a sentence attends to itself it's called self-attention — the core mechanism inside transformers.

Glossary

Attention
The mechanism that lets each word gather relevant context from the other words in a sentence.
Query
What a word is looking for when it scans the other words.
Key
A short label each word offers, advertising what it is about.
Value
The actual content a word contributes when it is attended to.
Match score
A number saying how well one word's query fits another word's key.
Softmax
A step that turns raw scores into percentages that add up to 100.
Attention weight
How much of a word's focus goes to a particular other word.
Self-attention
Attention where the words looking and the words looked at come from the same sentence.