Retrieval, Compression & Summarization — Context Engineering & Long-Context

A language model does not know your latest contract, support ticket, policy change, or design note unless that information is placed where the model can use it. Retrieval, compression, and summarization are the three practical moves that make this possible: find the right material, shrink it without breaking it, and express the result in a form a person can trust.

The problem: attention is useful but limited

A large language model works by reading a prompt and producing likely next words. The prompt may include the user’s question, instructions, examples, tool outputs, and source material. All of this must fit inside the model’s context window, which is the amount of text the model can consider at one time.

Real organizations have far more information than any prompt can hold. A single product manual may be longer than the context window. A customer case may include years of emails, logs, orders, and notes. If you paste everything in, the system becomes slow, expensive, and often worse at answering because the important evidence is buried in noise.

The core design task is therefore not simply to give the model more text. It is to give the model the right text, in the right shape, at the right moment. Retrieval, compression, and summarization are separate techniques, but in production they usually work together.

📌

First principle. The model can only use information that is in its context or reachable through tools at the time it answers.

Retrieval means finding useful evidence

Retrieval is the act of selecting relevant information from a larger store. A search engine retrieves web pages. A document system retrieves files. An artificial intelligence application retrieves passages, records, tickets, policies, or database rows that may help answer a question.

The simplest retrieval method is keyword search: match the words in the user’s query to words in documents. This works well when people use the same vocabulary. It fails more often when language varies, such as when a user asks about “leaving the company” but the policy says “employee offboarding.”

Many modern systems add semantic retrieval. Semantic means related to meaning, not just exact wording. The system turns text into embeddings, which are lists of numbers that represent meaning in a way computers can compare. Vector search then finds text whose embeddings are close to the query embedding. In practice, strong retrieval often combines keyword search, vector search, filters, and business rules.

Keyword search matches exact words; semantic search matches meaning through embeddings and vector comparison.

Chunking decides what can be retrieved

Before a system can retrieve from documents, it usually divides them into chunks. A chunk is a piece of source material that can be indexed and later placed into a prompt. Chunking sounds mechanical, but it strongly affects quality.

If chunks are too small, the system may retrieve a sentence without the surrounding explanation that makes it meaningful. If chunks are too large, the system may waste context on irrelevant material. Good chunks respect the structure of the source: headings, sections, tables, procedures, contract clauses, and conversation turns.

Metadata makes retrieval more precise. Metadata is data about data, such as document owner, product name, country, date, confidentiality label, or customer identifier. A system can use metadata to filter results before ranking them, which helps avoid showing a sales policy for the wrong region or a draft document that should not be used.

Chunk at natural boundaries, such as sections, clauses, or steps.
Keep enough surrounding text to preserve meaning.
Attach metadata that reflects permissions, freshness, domain, and source type.
Test retrieval with real questions, not only with clean sample queries.

Chunks that are too small lose meaning and chunks that are too large waste budget; the right size respects structure.

Compression means fitting more value into less space

Compression is reducing the amount of text while trying to preserve the parts that matter for the task. It is not the same as summarization, although summarization is one way to compress. Compression may remove duplicates, strip boilerplate, shorten logs, extract relevant fields, or keep only passages that answer a specific question.

Compression is necessary because retrieved material is often still too large. Ten search results may contain repeated definitions, legal disclaimers, old versions, and unrelated sections. A compression step can turn that mass into a smaller bundle of evidence before the model writes the final answer.

The main risk is silent loss. If compression removes a qualification, exception, date, or warning, the final answer may become confident and wrong. For high-stakes work, compressed text should preserve citations or pointers back to the original source so a person or system can inspect what was kept and what was dropped.

⚠️

Do not over-compress. A short answer is not better if it loses the condition that made the answer true.

Compression turns a noisy mass of retrieved results into a smaller bundle of evidence that keeps citations.

Summarization is a choice of purpose

Summarization is the act of producing a shorter expression of longer material. A good summary is not merely shorter; it is shorter for a purpose. An executive summary highlights decisions and risks. A support summary captures symptoms, actions, and next steps. A research summary preserves methods, findings, and limitations.

This purpose matters because different summaries of the same source can all be valid. A lawyer, engineer, product manager, and customer success lead may need different summaries from the same incident report. The system should be told the audience, the goal, the allowed length, and whether it should include uncertainty or open questions.

Summaries can be extractive or abstractive. Extractive summarization selects exact text from the source. Abstractive summarization rewrites the source in new words. Extractive summaries are easier to trace. Abstractive summaries are often easier to read, but they need stronger checks because the model may introduce wording that was not in the source.

How the pieces work together

A common pattern is retrieval-augmented generation, often shortened to RAG. Retrieval-augmented generation means the system retrieves relevant sources and gives them to a language model so the model can generate an answer grounded in those sources. Grounded means the answer is tied to supplied evidence rather than relying only on the model’s training.

A practical RAG system usually has two phases. In the preparation phase, documents are cleaned, chunked, labeled with metadata, embedded, and indexed. In the answering phase, the user’s request is interpreted, relevant chunks are retrieved, the best evidence is compressed, and the model writes an answer with citations or source references.

Microsoft Foundry can be used to build and evaluate these kinds of applications, while Foundry Tools can provide capabilities such as document processing, search, language understanding, and content safety. Identity and access decisions should use Microsoft Entra ID so retrieval respects who the user is and what they are allowed to see.

Retrieval-augmented generation retrieves and compresses relevant chunks, then grounds the model's answer in them.

Decisions leaders and practitioners must make

Retrieval, compression, and summarization are not only engineering details. They encode choices about cost, latency, trust, permissions, and accountability. A fast system may retrieve fewer sources. A cautious system may include citations, refuse uncertain answers, or ask a clarifying question before proceeding.

Leaders need to decide which use cases justify automation and which require human review. Practitioners need to decide how to chunk sources, which search methods to combine, how much compression to allow, and how to handle conflicting evidence. Security teams need to ensure the system retrieves only what the user is permitted to access.

The most important operational question is what happens when the system is unsure. A mature design does not force every request into a polished answer. It can say that the sources disagree, that no current source was found, or that a human should review the case.

Optimize for the business outcome, not for the longest possible prompt.
Treat permissions as part of retrieval, not as a user-interface feature added later.
Set rules for uncertainty, missing evidence, stale sources, and conflicting sources.
Measure cost and latency alongside answer quality.
Decide where citations are mandatory and where a short unsourced summary is acceptable.

Measuring quality and risk

A system can fail at several points. Retrieval may miss the right document. Ranking may put weak evidence above strong evidence. Compression may remove a crucial exception. Summarization may blur uncertainty. Generation may state a claim that the sources do not support.

Quality measurement should therefore look at the pipeline, not only the final answer. Useful tests include whether the correct source was retrieved, whether the answer used that source, whether citations point to the right place, whether the summary preserved key facts, and whether the system refused when evidence was missing.

Evaluation sets should contain real examples: common questions, rare edge cases, ambiguous requests, outdated documents, permission-sensitive content, and intentionally conflicting sources. The goal is not to prove the system is perfect. The goal is to learn where it is reliable, where it needs guardrails, and where a human must remain in the loop.

✅

Best signal. The strongest evidence of quality is not a fluent answer but an answer that can be traced back to the right source.

Key takeaways

Retrieval finds evidence; compression fits evidence into limited context; summarization expresses it for a purpose.
Chunking and metadata shape what a system can retrieve and who is allowed to see it.
Compression improves efficiency but can create risk if it removes conditions, exceptions, or source traceability.
Summaries should be designed for a specific audience, task, and level of uncertainty.
A reliable retrieval-augmented generation system measures each step, not just the final answer.
Access control, citations, uncertainty handling, cost, and latency are design decisions, not afterthoughts.

Glossary

Context window: The amount of text a language model can consider at one time when producing an answer.
Retrieval: The process of finding relevant information from a larger collection so it can be used for a task.
Chunk: A piece of source material that has been separated, indexed, and made available for retrieval.
Metadata: Data that describes other data, such as source, date, owner, region, permission label, or document type.
Embedding: A numerical representation of text that helps computers compare meaning rather than only exact words.
Vector search: A search method that finds items with embeddings that are close to the embedding of a query.
Compression: The process of reducing text or data while preserving the parts that matter for the task.
Summarization: The process of producing a shorter version of source material for a specific audience or purpose.
Retrieval-augmented generation: A pattern in which a system retrieves relevant sources and gives them to a language model so it can generate a grounded answer.