Temperature & Sampling — Prompting & Model Choice

A language model does not write a sentence in one sweep. It chooses one small piece of text, then another, then another, each time asking: given everything so far, what should come next? Temperature and sampling are the controls that decide whether the model usually picks the safest continuation, sometimes tries a less obvious one, or wanders into nonsense. They are simple knobs, but they sit on top of a precise process: turn scores into probabilities, choose one candidate, append it, and repeat.

The next-token problem

A large language model, often shortened to LLM, works with tokens. A token is a small chunk of text, such as a word, part of a word, a number, or a punctuation mark. When you send a prompt to the model, the model reads the tokens in that prompt and produces a score for many possible next tokens.

Those scores are not yet a final answer. They are more like preferences. In one context, the model may strongly prefer the token afternoon, weakly prefer evening, and barely consider banana. The system then turns these scores into probabilities and chooses one token. After that token is added, the model repeats the same process for the next token.

This matters because every answer is a chain of many small choices. A harmless-looking setting can affect hundreds or thousands of decisions in one response.

Scores, probabilities, and why the top answer is not always chosen

The model first produces raw scores, sometimes called logits. A logit is just a number saying how much the model favors one possible next token compared with others. Higher is more likely, but a logit by itself is not a probability.

A mathematical step called softmax converts logits into probabilities that add up to 100 percent. If one token has probability 70 percent and another has probability 20 percent, the first is more likely, but the second is not impossible.

If the system always chose the highest-probability token, the model would behave more predictably, but it could also become rigid, repetitive, or unable to explore alternative phrasings. Sampling means choosing according to the probability distribution rather than always taking the single top option.

📌

Probability is not truth. A high-probability token is the model's likely continuation, not a guarantee that the resulting statement is correct.

Raw logits become probabilities, then one token is sampled from that distribution.

Temperature changes the shape of choice

Temperature is a number applied before sampling. It changes how sharp or flat the probability distribution is. A lower temperature makes the strongest candidates stronger and the weaker candidates weaker. A higher temperature spreads probability more evenly across candidates.

At very low temperature, the model tends to choose the obvious next token. This can be useful for extraction, classification, code transformation, and other tasks where consistency matters. At higher temperature, the model is more willing to choose surprising tokens. This can help with brainstorming, naming, drafting, and creative exploration.

Temperature does not add knowledge to the model. It only changes how boldly the model samples from what it already considers possible.

Temperature shifts sampling from predictable to creative.

Sampling methods narrow the candidate set

Temperature reshapes probabilities, but sampling methods decide which candidates are eligible to be chosen. The most common controls are top-k and top-p sampling.

Top-k sampling keeps only the k highest-probability tokens. If k is 50, the sampler ignores every token outside the top 50 for that step. This prevents extremely unlikely tokens from appearing, but it uses the same fixed count no matter how confident or uncertain the model is.

Top-p sampling, also called nucleus sampling, keeps the smallest set of candidates whose probabilities add up to a chosen threshold, such as 90 percent. When the model is confident, that set may be small. When the model is uncertain, it may be larger. For that reason, top-p often adapts better to changing contexts.

Many systems combine temperature with top-p or top-k. The exact order and defaults can vary by provider, so production systems should record the full set of generation parameters rather than only the temperature.

Greedy decoding chooses the highest-probability token every time.
Temperature changes how concentrated the probabilities are.
Top-k limits choice to a fixed number of candidates.
Top-p limits choice to a probability mass that can grow or shrink at each step.

Top-k keeps a fixed count; top-p adapts the set to the model's confidence.

Determinism is more fragile than it looks

People often expect temperature 0 to mean fully repeatable output. In practice, it usually means something closer to choose the most likely next token. That is more deterministic than sampling at higher temperature, but it may not be perfectly repeatable across all systems and times.

There are several reasons. Two tokens may have nearly identical scores. Hardware and software can handle tiny numerical differences differently. A provider may update the model, tokenizer, safety layer, or serving system. Even if the prompt is unchanged, these hidden changes can alter the final text.

Some systems offer a seed, which is a number used to initialize a pseudo-random process so that sampling can be repeated more closely. A seed helps with reproducibility, but it is not a complete contract unless the model version, parameters, prompt, and serving behavior are also held steady.

⚠️

Do not test by vibes. For important workflows, evaluate outputs with fixed prompts, fixed parameters, version tracking, and task-specific checks.

How settings change real applications

The right temperature is not a moral choice between boring and creative. It is an engineering choice about variance. Variance means how much outputs differ from run to run. Some products benefit from variance; others are harmed by it.

For customer support routing, data extraction, policy checks, summarization with strict format, and code edits, lower variance is usually safer. The goal is to reduce surprises and make failures easier to detect. For ideation, alternative explanations, style exploration, synthetic examples, and first drafts, some variance may be useful because the system is meant to broaden the search.

A leader or practitioner should decide what the application is optimizing for: correctness, consistency, novelty, user delight, cost, latency, or auditability. Temperature and sampling settings should follow that decision, not the other way around.

Let the task's need for consistency drive the temperature setting.

Practical starting points

There is no universal best temperature. A useful approach is to start with the task's tolerance for variation, then test a small range. Keep prompts, model version, and evaluation data stable while changing one setting at a time.

For structured or factual tasks, start low. For open-ended drafting, start in the middle. For playful or highly creative work, try higher values, but watch for drift, unsupported claims, and format breakage. If the answer must follow a schema, sampling freedom can work against you unless the system also enforces the schema.

When using an application programming interface, or API, store the generation parameters with the output. That record should include model name or version, temperature, top-p or top-k if used, maximum output length, seed if available, and the prompt template. Without these details, debugging becomes guesswork.

Define the job and the cost of a bad answer.
Pick a low, medium, or high variance target.
Run the same evaluation set across several settings.
Measure task outcomes, not just whether the prose feels good.
Record the exact parameters used in production.

A mental model to keep

Think of the model as offering a ranked menu for the next token. Temperature changes how adventurous the chooser feels. Top-k and top-p decide which menu items remain available. The sampler then picks one item and the whole process starts again.

This mental model prevents two common mistakes. The first is treating temperature as a quality dial, as if turning it up makes the model smarter. The second is treating low temperature as a guarantee of truth. Temperature and sampling shape expression and variation; they do not replace grounding, validation, or good product design.

Key takeaways

Language models generate text one token at a time by scoring possible next tokens.
Temperature reshapes probabilities: lower values concentrate choices, higher values spread them out.
Sampling methods such as top-k and top-p limit which tokens are eligible to be chosen.
Low temperature improves consistency but does not guarantee correctness or perfect repeatability.
Choose settings based on the application's tolerance for variance, risk, and need for novelty.
Record model versions and generation parameters so outputs can be tested and debugged.

Glossary

Token: A small unit of text, such as a word, part of a word, number, or punctuation mark, that a language model reads and writes.
Large language model: A machine learning system trained to predict and generate text by learning patterns from large collections of written material.
Logit: A raw model score that indicates how strongly the model favors a possible next token before probabilities are calculated.
Softmax: A mathematical function that converts raw scores into probabilities that add up to 100 percent.
Temperature: A generation setting that makes token probabilities sharper at lower values or flatter at higher values.
Sampling: The process of choosing a token according to probabilities rather than always selecting the highest-scoring option.
Top-k sampling: A sampling method that allows only the k highest-probability tokens to be considered at each generation step.
Top-p sampling: A sampling method that allows the smallest set of tokens whose probabilities add up to a chosen threshold.
Seed: A number used to initialize a pseudo-random process so repeated runs can be made more similar under the same conditions.