Tokens, Cost & Latency — Prompting & Model Choice

A large language model does not see a question the way a person does. It sees a sequence of small text pieces called tokens, turns them into numbers, and predicts more tokens one by one. That simple fact sits underneath three everyday concerns: how much an artificial intelligence application costs, how quickly it responds, and how much conversation or source material it can handle at once.

A token is the model's working unit

A token is a piece of text used by a model as its basic input and output unit. A token may be a whole short word, part of a long word, a punctuation mark, a space pattern, or a fragment from another writing system. The word cat is often one token. The word internationalization may be split into several tokens. An emoji or a line break can also consume tokens.

The split is done by a tokenizer, which is a program that converts human-readable text into token identifiers the model can process. The model does not bill, remember, or generate by characters or pages. It works through token sequences.

A rough English rule of thumb is that one token is about three quarters of a word, or about four characters, but this is only a planning estimate. Code, tables, legal text, and non-English languages can behave very differently.

Cost starts with counted tokens

Most hosted model services price usage by tokens. The bill usually has at least two sides: input tokens, which are the tokens you send to the model, and output tokens, which are the tokens the model generates. Output tokens often cost more because generating them requires repeated model work.

A prompt is everything you send to the model for one request: the user's message, instructions, examples, retrieved documents, tool results, and often some hidden system guidance. A completion is the model's generated answer. The cost of a request is mainly the prompt tokens plus the completion tokens, multiplied by the relevant prices.

This makes cost sensitive to product design. A chatbot that resends a full conversation on every turn may become expensive as the conversation grows. A document assistant that includes ten pages when one paragraph would do pays for the unused nine pages every time.

Input cost comes from every token sent in the prompt, including repeated conversation history and retrieved material.
Output cost comes from every token generated, including verbose answers, explanations, and structured formats.
Operational cost includes retries, failed requests, safety checks, logging, evaluation runs, and background jobs.

A request's cost is input tokens plus output tokens, and output usually costs more.

Latency is not one delay

Latency is the time a user waits for a system to respond. With language models, that wait has parts. There may be network time, queueing time, prompt processing time, time to first token, and then the time needed to stream the rest of the answer.

Time to first token is the delay before the first visible piece of the answer appears. After that, the model emits tokens in sequence. This matters because generation is partly serial: each new token depends on the tokens before it. A longer answer usually means a longer wait, even if the first token arrives quickly.

Throughput is the amount of work a system can complete in a period of time, often measured as requests per second or tokens per second. A system can feel fast for one user but still have poor throughput under load, or it can have high throughput while some individual requests wait in a queue.

The wait is network time, queueing, time to first token, then streaming the rest.

The context window is useful but not free

The context window is the maximum number of tokens a model can consider in one request. It includes the prompt and the completion together. If a model has a 100,000 token context window and you send 90,000 input tokens, there is less room left for the answer.

A larger context window lets an application include more history, more documents, or more examples. That can improve results when the added material is relevant. It can also make the model slower and more expensive, and it can dilute attention if the important facts are buried inside clutter.

Long context should not be treated as a replacement for information design. The better question is not how much can fit, but what the model truly needs for this task, in this turn, for this user.

⚠️

Bigger is not automatically better. A long prompt can raise cost and latency while making the answer worse if it contains irrelevant or conflicting material.

The context window is one shared budget for prompt and completion together.

The levers you actually control

You cannot change the physics of a model at request time, but you can shape the work you ask it to do. The highest-impact improvements often come from reducing unnecessary input, limiting unnecessary output, and choosing the smallest model that reliably meets the need.

Caching is one of the most practical tools. Caching means reusing a previous result or reusing processed prompt content when the same stable material appears again. For example, a long policy document that is unchanged across many requests may be handled more efficiently than resending and reprocessing it as if it were new each time.

Streaming is another useful design choice. Streaming does not necessarily make the model finish sooner, but it can make the product feel faster because the user sees the answer begin while the rest is still being generated.

Remove boilerplate instructions that do not change behavior.
Summarize or trim old conversation turns before they grow without bound.
Retrieve only the most relevant passages instead of attaching whole documents.
Set output limits and ask for concise formats when appropriate.
Use a smaller or faster model for simple tasks and reserve larger models for harder ones.
Cache stable prefixes, repeated lookups, and deterministic intermediate results.

✅

Best first move. Measure prompt tokens and output tokens separately before trying to optimize either one.

Shrink the work you ask for: trim input, limit output, pick a smaller model, and cache.

Real decisions this changes

For a product leader, tokens are not an implementation detail. They affect unit economics, user experience, capacity planning, and the shape of the product itself. A generous free-form assistant may be delightful in demos but costly in production if every turn carries a long history and produces a long answer.

For an engineer or architect, token design becomes part of system design. You decide how much history to keep, which documents to retrieve, where to summarize, when to stream, which calls can be batched, and what service-level objective to promise. A service-level objective is a stated target for reliability or speed, such as 95 percent of answers starting within two seconds.

These decisions are tradeoffs, not moral choices. A medical or legal review tool may accept higher latency and cost to include more evidence and produce careful citations. A customer support triage tool may prefer short prompts, fast classification, and escalation when confidence is low. The right token budget follows the job the product must do.

Measure before you tune

Token optimization is easy to overdo in the abstract and easy to underdo in production. The remedy is measurement. Record input tokens, output tokens, total latency, time to first token, model choice, retry count, cache hits, and whether the user or evaluator judged the answer successful.

Look at distributions, not just averages. A system with an average latency of two seconds may still have a painful tail if one in twenty users waits ten seconds. The long tail often reveals oversized prompts, queueing under load, slow retrieval, or answers that run past their useful length.

Finally, measure quality beside cost and speed. Cutting tokens is only an improvement if the task still succeeds. The goal is not the fewest possible tokens; it is the fewest tokens that produce the needed result at the speed and reliability the product requires.

Key takeaways

A token is the unit a language model reads, writes, counts, and usually bills for.
Cost is driven by both input tokens and output tokens, plus retries and supporting workflow calls.
Latency includes queueing, prompt processing, time to first token, and the time to generate the remaining tokens.
Large context windows are powerful, but irrelevant context can increase cost, slow responses, and reduce answer quality.
The main practical levers are trimming input, limiting output, choosing the right model, streaming, and caching.
Optimize against measured quality, cost, and latency together, not against token count alone.

Glossary

Token: A token is a small piece of text that a model uses as its basic unit for reading and writing.
Tokenizer: A tokenizer is the program that splits text into tokens and maps them to numeric identifiers.
Prompt: A prompt is all the information sent to the model for a request, including instructions, user text, history, and added context.
Completion: A completion is the text generated by the model in response to a prompt.
Context window: A context window is the maximum number of tokens a model can consider in a single request, including both input and output.
Latency: Latency is the time a user or system waits for a response.
Time to first token: Time to first token is the delay between sending a request and receiving the first generated token.
Throughput: Throughput is the amount of work a system completes in a given time, such as requests per second or tokens per second.
Caching: Caching is the practice of reusing stored results or processed content to avoid repeating the same work.