From Words to Numbers: Tokens and Embeddings

A language model looks like it reads and writes the way we do, but underneath it never sees a single letter. It only ever works with numbers. So the very first thing that happens to your message is a quiet translation: your words are chopped into small pieces, each piece is given a number, and each number is turned into a little list of numbers that captures what the piece means. Once you understand that translation, everything else a model does starts to make sense.

Why a computer needs numbers, not words

Imagine trying to do arithmetic with the letter 'k'. You can't — letters aren't quantities. Computers have the same problem: at their core they add and multiply numbers, and nothing else. A language model (a computer program trained to understand and produce text) is no exception. Before it can do anything with your sentence, that sentence has to become numbers.

So think of the model as a brilliant mathematician who doesn't speak your language. To work with your words, it first has to convert them into something it can calculate with. That conversion happens in two steps, and this article walks through both: first cutting your text into pieces called tokens, then turning each piece into a list of numbers called an embedding.

📌

The one-line version. Words become tokens, tokens become ID numbers, and ID numbers become embeddings — lists of numbers that stand for meaning.

Step one: chop the text into tokens

A token is just a small chunk of text — often a whole word, but sometimes a piece of one. The model has a fixed set of chunks it knows, and it breaks your sentence into those chunks. Common words like 'the' or 'cat' are usually a single token. A longer or rarer word like 'unbelievable' might be split into a few pieces, such as 'un', 'believ', and 'able'.

Why split words at all? Because it lets a model handle any word — even ones it has never seen — by building it out of familiar pieces, the same way you can sound out a new word syllable by syllable. Chopping text this way is called tokenizing, and the pieces it produces are the tokens the model actually works with.

Your sentence is split into small, known chunks called tokens.

Step two: give every token an ID number

Each token the model knows lives in a big fixed list called the vocabulary — think of it as the model's dictionary. Every entry in that dictionary has a position number, and that number is the token's ID. So the word 'cat' might be entry number 9,481, and 'dog' might be 12,006. The model doesn't care what the numbers are; it just needs a consistent ID for each token, like a coat-check ticket that always points to the same coat.

After this step, your sentence has become a short list of ID numbers — one per token. That's already progress: the model now has numbers. But an ID on its own carries no meaning. Nothing about the number 9,481 tells the model that a cat is an animal, or that it's similar to a kitten. For that, we need the second, richer kind of number.

'cat' → token → ID number (say 9,481)
'sat' → token → ID number (say 4,102)
A whole sentence → a short list of ID numbers

From an ID to meaning: the embedding

Here is the clever part. Instead of leaving 'cat' as the single lonely number 9,481, the model looks it up in a table and replaces it with a whole list of numbers — maybe a few hundred of them. That list is called an embedding, and each number in it is like a score for some hidden trait: how 'animal-like' the word is, how 'small', how 'furry', how 'formal', and so on. Nobody hand-labels these traits; the model learns them while training. The point is that meaning is no longer a single number — it's a rich profile.

A list of numbers like this is called a vector, and the number of entries in it is called its dimensions. You don't need the jargon to get the idea: an embedding is a detailed numeric fingerprint of a word's meaning. Two words with similar meanings end up with similar fingerprints.

What you read as a word, the model stores as a list of numbers that captures its meaning.

A map of meaning

Because similar words get similar fingerprints, you can picture every word as a dot on a giant map, where words that mean related things sit close together. On this 'map of meaning', 'cat' and 'kitten' would be near neighbours; 'dog' and 'puppy' would be close by too; and 'cat' would be far away from 'bicycle'. The model builds this map itself, purely from reading enormous amounts of text and noticing which words show up in similar situations.

This is why embeddings are so powerful. Distance on the map means something: nearby fingerprints mean similar meanings. That single idea — meaning turned into position — is the foundation everything else in this series builds on.

Words with related meanings end up close together in the model's map of meaning.

✅

The key idea. An embedding turns meaning into a location — similar meanings sit near each other, and the model can measure that closeness with plain arithmetic.

Why this matters for everything that follows

Once your words are embeddings, the model finally has something it can compute with. Every later step — deciding which earlier words matter, predicting the next word, holding a long conversation — is really just arithmetic performed on these numeric fingerprints. When a model seems to 'understand' that a sentence is about pets, what's actually happening is that the numbers for those words are close together on the map of meaning.

In the next article we'll see the first thing the model does with these fingerprints: a mechanism called attention, where each word looks at the others and decides which ones matter most for what comes next. It all runs on the numbers we just built.

A practical bonus: tokens measure length and cost

Tokens aren't only an internal detail — they're also the unit that decides how much text a model can consider at once and, usually, what you pay to use it. The amount of text a model can keep in mind at one time is called its context window, and it's measured in tokens, not words. Pricing for hosted models is typically per token too.

A rough rule of thumb in English: one token is about four characters, so a hundred tokens is roughly seventy-five words. Knowing that helps you reason about limits and cost without any mystery — long inputs simply mean more tokens.

Context window = how much the model can consider at once, counted in tokens.
Cost is usually per token, for both what you send and what you get back.
Rough guide: ~4 characters per token; ~75 words per 100 tokens.

Key takeaways

A language model only works with numbers, so your words are translated before anything else happens.
Text is first cut into tokens — small chunks that are often whole words but sometimes word-pieces.
Each token gets an ID number from the model's fixed vocabulary, but an ID alone carries no meaning.
An embedding replaces that ID with a list of numbers — a numeric fingerprint of the word's meaning.
Similar meanings get similar fingerprints, so words form a 'map of meaning' where closeness equals similarity.
Tokens also measure length and cost: the context window and pricing are counted in tokens, not words.

Glossary

Language model: A computer program trained to understand and produce human language.
Token: A small chunk of text — often a whole word, sometimes a piece of one — that the model treats as a unit.
Tokenizing: Cutting a piece of text into tokens the model recognises.
Vocabulary: The fixed list of all tokens a model knows, like its dictionary.
Token ID: The position number of a token in the vocabulary, used to refer to it.
Embedding: A list of numbers that stands for a token's meaning — a numeric fingerprint.
Vector: Simply a list of numbers; an embedding is one kind of vector.
Context window: How much text a model can consider at once, measured in tokens.