Context Evaluation and Lost-in-the-Middle — Context Engineering & Long-Context

A large language model (LLM) does not remember your business, read your files, or know your customer unless that information is placed where the model can use it. The practical question is not only whether the right information was retrieved, but whether the model actually noticed it, trusted it, and used it in the answer. Context evaluation is the discipline of testing that chain. Lost-in-the-middle is one of the failure modes it exposes: a model can perform well when the key fact is near the start or end of a long context, yet miss the same fact when it is buried in the middle.

What context means from first principles

A prompt is the input sent to a model. It may contain the user’s question, system instructions, conversation history, retrieved documents, tool results, and formatting rules. The model receives all of this as one sequence of tokens, where a token is a small unit of text such as a word fragment, word, or punctuation mark.

The context window is the maximum number of tokens the model can consider at one time. Larger context windows are useful because they can hold more evidence, but they are not magic. Adding more text increases the chance that important details compete with irrelevant or repetitive details.

A useful way to think about context is as working memory, not storage. Storage can contain millions of documents. Context is the small working set placed in front of the model for this answer, right now.

📌

Working memory. A long context window gives the model more room, but it does not guarantee that every fact receives equal attention.

Why retrieval is only half the job

Many applications use retrieval-augmented generation (RAG), a pattern in which the system searches external content and inserts selected passages into the prompt before the model answers. RAG helps because the model can ground its answer in current or private information rather than relying only on patterns learned during training.

But a RAG system can fail in two different places. The retriever may fail to bring back the right passage. Or the retriever may bring back the right passage and the model may still fail to use it. Context evaluation focuses on the second problem as well as the first.

This distinction matters because a retrieval dashboard may say the correct document was returned, while a user still receives a wrong answer. The evidence arrived, but it did not affect the response.

Retrieval asks whether the right evidence was found.
Context assembly asks whether the evidence was placed clearly and compactly.
Generation asks whether the answer used the evidence correctly.

A RAG system can fail because the retriever misses the passage, or because the model ignores a passage it was given.

The lost-in-the-middle effect

Lost-in-the-middle describes a pattern in which a model is more likely to use information near the beginning or end of a long context than information placed in the middle. The model may still have the middle text inside its context window, but its answer behaves as if that text was weaker, less visible, or less influential.

This happens for several reasons. The beginning often contains instructions and framing, so it can shape the model’s interpretation. The end is close to the point where the model starts generating its answer, so it can be especially salient. The middle can become a crowded zone where many passages look similar and none stands out.

The effect is not a universal law and it varies by model, prompt design, and task. Still, it is common enough that teams should test for it instead of assuming that longer context automatically means better accuracy.

⚠️

Long is not safe. A 100,000-token context can still fail if the decisive sentence is surrounded by distracting material.

Models tend to use information at the beginning or end of a long context more than information buried in the middle.

What context evaluation measures

Context evaluation is the systematic testing of whether the supplied context is sufficient, relevant, readable by the model, and actually reflected in the answer. It treats the prompt as an engineered object that can be measured and improved.

A good context evaluation separates several questions. Did the system include the necessary evidence? Did it exclude unnecessary evidence? Was the evidence placed where the model could use it? Did the final answer cite, paraphrase, or reason from that evidence rather than from unsupported assumptions?

The best evaluations use tasks with known answers. For example, a test case may ask for a policy exception that appears in exactly one paragraph. The same paragraph can be placed at the beginning, middle, and end of the context to see whether position changes the answer.

Context evaluation checks that necessary evidence is included, noise is excluded, evidence is placed where it helps, and the answer reflects it.

How to design a practical test

Start with a small set of realistic questions where the correct answer is known and traceable to specific evidence. Synthetic tests are useful for isolating position effects, but they should be paired with real examples from support tickets, policies, contracts, medical guidelines, engineering runbooks, or whatever domain the system serves.

For each test case, record the source passage, the expected answer, the passages retrieved, the final assembled prompt, and the model’s response. Without the assembled prompt, teams often argue about symptoms while missing the cause.

Then vary one thing at a time. Move the decisive passage from the top to the middle to the bottom. Add similar but irrelevant passages. Change chunk size, ranking, and summarization. Compare outcomes across models and prompt templates. Tools in platforms such as Microsoft Foundry or open-source evaluation harnesses can help automate the runs, but the test design matters more than the tool.

Choose questions with verifiable answers.
Identify the exact evidence needed to answer each question.
Generate prompt variants that change evidence position and noise level.
Score whether the answer is correct, grounded, complete, and consistent.
Review failures by reading the actual assembled prompt.

✅

Test the prompt. The assembled prompt is the real input to the model, so it must be logged and evaluated like any other production artifact.

Patterns that reduce middle loss

The simplest mitigation is to keep context smaller and sharper. Remove boilerplate, duplicate text, navigation fragments, irrelevant metadata, and passages that only loosely match the question. A model usually benefits more from five strong passages than from fifty weakly related ones.

Place the most important evidence deliberately. Many teams put a short evidence summary before the detailed passages, then preserve the original passages for grounding. Others repeat key constraints near the answer instruction, especially when safety, policy, or contractual terms are involved. Repetition should be careful; repeating everything creates more noise.

Chunking also matters. A chunk is a unit of content prepared for retrieval, such as a paragraph, section, or page. Chunks that are too small lose context; chunks that are too large bury the answer. Good chunking keeps a useful idea together and includes enough surrounding text for interpretation.

Rank evidence by likely usefulness, not merely by search similarity.
Put decisive facts near clear labels such as “Relevant policy” or “Required constraint.”
Summarize long evidence before including full supporting text.
Use reranking to promote the passages most likely to answer the exact question.
Measure accuracy after every change rather than relying on intuition.

⚠️

Do not stuff. Adding every possible document to the prompt often hides the answer instead of protecting it.

Reduce lost-in-the-middle by trimming noise, summarizing evidence up front, placing it deliberately, and repeating key constraints.

The decision leaders and practitioners face

Context evaluation turns an abstract model discussion into an operating decision. If failures come from missing retrieval, invest in better indexing, metadata, permissions, and search ranking. If failures come from lost-in-the-middle, invest in prompt assembly, chunk design, reranking, and evidence compression. If failures come from reasoning over correct evidence, invest in task decomposition, tool use, or a different model.

Leaders should ask for accuracy broken down by evidence position, context length, noise level, and source type. A single average score can hide the exact failure that will appear in production, especially when users ask broad questions that pull in long contexts.

Practitioners should treat context as a budget. Every token spent on weak evidence is a token that competes with strong evidence. The goal is not to fill the context window; the goal is to make the right facts easy for the model to find, weigh, and use.

Key takeaways

Context is the model’s working memory for one answer, not a guarantee that every included fact will be used.
Lost-in-the-middle means important information can be ignored when it sits inside a long, crowded prompt.
Retrieval success and answer success are different; the right passage can be retrieved and still not shape the response.
Context evaluation should test evidence position, noise, coverage, and whether the final answer is grounded in the supplied facts.
Better results often come from sharper context, better chunking, reranking, and deliberate evidence placement, not simply larger context windows.

Glossary

Chunk: A chunk is a prepared unit of content, such as a paragraph or section, that can be retrieved and placed into a prompt.
Context evaluation: Context evaluation is the practice of testing whether the information placed in a prompt is relevant, sufficient, clear, and used correctly by the model.
Context window: A context window is the maximum amount of text, measured in tokens, that a model can consider at one time.
Grounding: Grounding means tying an answer to supplied evidence rather than unsupported model assumptions.
Large language model: A large language model is a system trained on large amounts of text to predict and generate language.
Lost-in-the-middle: Lost-in-the-middle is the tendency for a model to miss or underuse important information placed in the middle of a long context.
Prompt: A prompt is the complete input sent to a model, including the user question, instructions, retrieved evidence, and any conversation history.
Retrieval-augmented generation: Retrieval-augmented generation is a pattern where a system searches external content and inserts relevant passages into the prompt before the model answers.
Token: A token is a small unit of text, such as a word, word fragment, number, or punctuation mark, used by a model to process language.