Tokens and Context Windows — Meet the AI Agent

A language model does not read a document the way a person does. It receives a long line of symbols, broken into small pieces called tokens, and uses those pieces to predict what should come next. The context window is the model's temporary field of view: the prompt, instructions, examples, retrieved facts, conversation history, tool results, and draft answer that can be considered at one time. Most surprises in real systems trace back to this simple constraint: the model can only reason over what fits, and fitting more is not the same as understanding more.

From words to tokens

A token is a chunk of text that a model treats as one unit. A token may be a whole word, part of a word, a space plus a word, punctuation, a number fragment, or a character from a non-Latin writing system. The exact split is chosen by a tokenizer, which is the software that converts text into tokens before the model sees it.

Tokenization exists because computers need discrete inputs. Human language is messy: new names appear, words combine, spelling varies, and many languages do not separate words with spaces. A tokenizer gives the model a practical alphabet of reusable pieces. The model then works with token identifiers, not with letters on a page.

A rough English rule is that one token is often about three or four characters, or about three quarters of a word. That rule is only a convenience. Code, tables, legal citations, emojis, and multilingual text can use tokens very differently. Counting words is therefore a weak way to predict cost or fit.

A tokenizer splits text into token IDs before the model ever sees it.

Why tokens matter

Tokens are the basic accounting unit for many language model systems. They affect how much input fits, how much output can be produced, how long a request takes, and often how much the request costs. If a system bills per token, both the prompt tokens and the generated answer tokens matter.

Tokens also affect quality. If important facts are outside the context window, the model cannot use them. If the prompt is full of duplicated history, boilerplate, or irrelevant retrieved passages, the useful information competes with noise. A bigger prompt can make the answer worse when it distracts the model from the task.

This is why token budgeting is not only a finance exercise. It is an engineering design habit. Good systems decide what deserves space: the user's current request, durable instructions, essential examples, relevant data, and enough history to preserve intent.

📌

Two counts. Input tokens are what you send to the model, and output tokens are what the model generates in response.

The context window

The context window is the maximum number of tokens the model can consider in one request. If a model has a 32,000-token context window, the sum of the prompt and the answer must fit within that limit. A long prompt leaves less room for the answer. A requested long answer leaves less room for source material.

The window is temporary. It is not the model's memory in the human sense. Unless the application sends earlier conversation turns again, the model does not automatically have them available. Some products store summaries or facts between sessions, but that is an application feature built around the model, not the same thing as the context window.

Think of the context window as a workbench. You can place instructions, documents, notes, and the partially written answer on it. The model can work only with what is on the bench. A larger bench helps, but it still requires arrangement.

The context window is a shared bench holding everything a request needs.

What happens near the limit

When the input is too large, an application must do something before the model can run. It may reject the request, trim older conversation, shorten documents, summarize content, or retrieve only selected passages. If this happens silently, users can get answers that sound confident but are based on missing material.

Even before a hard limit is reached, very long prompts create practical problems. Latency can rise because there is more to process. Cost can rise because there are more tokens. The chance of including stale or irrelevant context can rise because large prompts often accumulate material rather than choose it.

There is also a quality issue sometimes called attention dilution. Modern models can handle long contexts impressively well, but they are not equally sensitive to every token in every position for every task. Important details buried in a large bundle may be overlooked, especially when the prompt contains conflicting or repetitive information.

⚠️

Long is not free. A larger context window is useful, but it does not remove the need to select, order, and test the information you provide.

How models use context

At inference time, which means when the model is generating an answer, the model repeatedly predicts the next token. Each new token is appended to the context, and the next prediction uses the updated context. This continues until the model stops, reaches an output limit, or is interrupted by the application.

A key mechanism is attention. Attention is the model's learned way of relating tokens to other tokens in the same context. It helps the model connect a pronoun to a name, a question to a relevant paragraph, or an instruction to a requested format. Attention is not the same as comprehension, but it is one reason the model can use surrounding text rather than generating from a single word at a time.

Because the answer is built token by token, generation is sequential. The model cannot know its final wording in advance. It can follow patterns and constraints, but long outputs may drift unless the prompt provides a clear structure, the application checks the result, or the task is broken into smaller steps.

The model predicts one token, appends it, and repeats until it stops.

Design choices for real systems

Leaders and practitioners face a recurring decision: should the system send more context, better context, or a smaller task? The answer depends on the job. A contract review may need long source documents. A customer support bot may need only the current issue, account facts, and the two or three most relevant policy passages. A coding assistant may need selected files, recent edits, and test failures rather than an entire repository.

The most reliable systems treat the context window as a scarce resource. They rank retrieved information, remove duplicates, preserve source labels, and keep instructions short and stable. They also measure what happens when context is changed. If removing a passage does not hurt the answer, that passage may not deserve space.

This choice also affects governance. Sensitive data placed in the prompt is exposed to the model request path and may appear in logs, traces, or downstream tool calls depending on the system design. Token budgeting and data minimization often support the same goal: send only what is needed for the task.

Set a maximum input budget for each task type, not just for the whole application.
Reserve output space so the model does not run out of room mid-answer.
Prefer relevant excerpts with citations over whole documents when the task allows it.
Summarize conversation history only when the summary is tested against real follow-up questions.
Track latency, cost, and answer quality as separate measures because they do not move together.

Practical rules of thumb

Start with the user's current request and the minimum instructions required to answer it well. Add context only when it changes the result. If the model needs facts, retrieve the most relevant facts. If it needs style, provide a short example. If it needs a process, give steps. Do not make the prompt a storage closet.

Place the most important instructions and facts where the model is likely to use them: clear, explicit, and close to the task. Remove contradictions. If a system instruction says one thing and a retrieved document says another, decide which should win before the request is sent.

Finally, test with cases that stress the window: long documents, repeated conversation turns, multilingual text, tables, code, and users who ask for lengthy outputs. Token problems often hide in demos because demos are short.

Key takeaways

A token is the unit a language model reads and writes; it is not always a word.
The context window is the temporary token space shared by instructions, data, history, and the answer.
Longer context can help, but irrelevant or duplicated context can raise cost, latency, and error risk.
Applications must decide what to include, trim, summarize, or retrieve before each model request.
Reserve room for output tokens, or the model may stop before completing the answer.
Treat context as a design resource, not as a dumping ground.

Add context only when it changes the result, not to fill space.

Glossary

Token: A small text piece, such as a word fragment, punctuation mark, or character sequence, that a model processes as one unit.
Tokenizer: Software that converts text into tokens before the text is sent to a model.
Context window: The maximum number of tokens a model can consider in a single request, including both input and generated output.
Prompt: The full input sent to a model, including instructions, user requests, examples, history, and supporting data.
Inference: The process of running a trained model to generate an answer or prediction.
Attention: A model mechanism that relates tokens in the context to other tokens so the model can use surrounding information.
Latency: The time it takes for a system to respond to a request.
Retrieval: The process of selecting relevant information from an external source and adding it to the prompt.