Chunking & Indexing — Building AI Apps: Retrieval

A search system does not understand a library by staring at whole books. It needs smaller, well-labeled pieces it can compare, rank, and return at the moment someone asks a question. Chunking is the act of cutting source material into those pieces. Indexing is the act of arranging the pieces so they can be found quickly and reliably. Get either step wrong and even the best model or search engine will sound confused, miss obvious answers, or quote the right document for the wrong reason.

The basic problem: finding the right context

Imagine a person asks, “What is our refund policy for annual plans?” The answer may be buried in a 60-page handbook, a help-center article, or a contract addendum. A computer can scan all of that text, but scanning everything for every question is slow, expensive, and often noisy.

The practical solution is to prepare the material before questions arrive. We break long sources into smaller units, store those units with useful labels, and build a structure that makes likely matches easy to retrieve. The retrieved text then becomes the context for a search result, an answer, or a retrieval-augmented generation (RAG) system, where a language model writes using retrieved source material.

Chunking and indexing are therefore not clerical preprocessing steps. They shape what the system can know, what it can cite, and what it is likely to overlook.

What a chunk really is

A chunk is a piece of source content treated as one retrievable unit. It might be a paragraph, a section, a table row, a slide note, a support ticket, or a carefully sized window of text. The chunk is the unit the system compares to a question and returns as evidence.

Good chunks preserve meaning. A chunk that is too small may say “this limit applies” without saying what limit. A chunk that is too large may contain the answer, three unrelated topics, and enough extra words to confuse ranking. The goal is not to make every piece the same size. The goal is to make each piece useful when read alone or with its immediate neighbors.

Most systems measure text length in tokens, which are small pieces of text used by language models and search tools. A token may be a word, part of a word, or punctuation. Token limits matter because they set how much text can be embedded, indexed, retrieved, and later passed into a model.

⚠️

Size is not meaning. A 500-token chunk can be excellent or terrible depending on whether it follows the structure of the source.

Common chunking strategies

The simplest strategy is fixed-size chunking: cut text every set number of tokens or characters. It is easy to automate and predictable to operate, but it can split a sentence, table, or argument at the worst possible place.

Structure-aware chunking follows the shape of the source. It keeps headings with the paragraphs below them, treats list items as related, and avoids separating a table caption from the table. This usually produces better retrieval because the chunk reflects how a human author organized meaning.

Semantic chunking tries to split where the topic changes. It may use sentence boundaries, headings, similarity between passages, or a language model. This can work well for messy material, but it adds complexity and can be harder to test because the cuts are less predictable.

Many production systems use overlap, where the end of one chunk is repeated at the start of the next. Overlap reduces the chance that an answer is split across a boundary. Too much overlap, however, fills the index with near-duplicates and can crowd out more useful results.

Use fixed-size chunking for simple logs, transcripts, or uniform text when speed matters more than nuance.
Use structure-aware chunking for manuals, policies, knowledge bases, and reports with headings and sections.
Use semantic chunking when sources are inconsistent or when topic boundaries matter more than document formatting.
Use overlap when answers often span paragraphs, but measure whether duplicates hurt result quality.

Text can be split by fixed size, by document structure, or where the topic changes.

What indexing adds

Indexing turns a collection of chunks into something searchable. Without an index, the system would have to compare every question with every chunk each time. That may work for a dozen pages, but it collapses at thousands or millions of chunks.

An index stores signals that help the system narrow the search. A keyword index tracks exact words and phrases. A vector index stores embeddings, which are numerical representations of meaning. A metadata index stores labels such as source, date, department, product, author, permissions, and version.

Modern retrieval often combines these signals. Exact keyword search is strong when the user names a product code, statute, or error message. Vector search is strong when the user describes an idea in different words than the source uses. Metadata filters keep the system from retrieving expired, unauthorized, or irrelevant material.

Retrieval combines a keyword index, a vector index, and a metadata index.

Retrieval quality: recall, precision, and ranking

Once chunks are indexed, the system must decide what to return for a query. Three ideas matter. Recall is whether the system finds the relevant chunks at all. Precision is whether the returned chunks are mostly useful rather than clutter. Ranking is whether the best chunks appear near the top.

Chunking affects all three. If chunks are too large, recall may look good because the answer is somewhere inside, but precision suffers because each result carries irrelevant text. If chunks are too small, precision may look good at first, but recall suffers because the system retrieves fragments without enough context to prove the answer.

Indexing affects all three as well. A weak vector index may miss paraphrased questions. A weak keyword index may miss exact terms hidden in scanned documents or tables. Poor metadata can return old policies, content for the wrong region, or documents the user should not see.

The right test is not whether the system can answer one polished demo question. The right test is a set of real questions, expected source passages, and judgments about whether the retrieved chunks would let a careful person answer correctly.

📌

Test retrieval first. Before judging the final answer, inspect the chunks the system retrieved and ask whether the evidence was good enough.

Small chunks favor precision but lose context; large chunks favor recall but add clutter.

Operational details that matter

Chunking and indexing are ongoing operations, not one-time setup. Source material changes. Policies expire. Product names are reused. Access permissions shift. If the index does not track these changes, the system becomes confidently stale.

Each chunk should carry metadata that makes it governable: where it came from, when it was created, when it was indexed, who can read it, what version it belongs to, and whether it is authoritative. This metadata is not decoration. It is how the retrieval layer enforces freshness, scope, and trust.

Tables, images, code, and scanned documents need special care. A table may need to be converted into text that preserves row and column meaning. An image may need a caption or optical character recognition, which is the process of turning visible text in an image into machine-readable text. Code may need chunks that respect functions and files rather than paragraph length.

Re-indexing must also be planned. Some systems rebuild the whole index on a schedule. Others update only changed documents. Incremental updates are efficient, but they require careful deletion and version handling so old chunks do not linger beside new ones.

Each chunk should carry source and version, dates, permissions, and whether it is authoritative.

Decisions leaders and practitioners must make

Chunking and indexing choices are design decisions with cost, risk, and user experience consequences. Smaller chunks may improve focus but increase index size and retrieval volume. Larger chunks may reduce storage and simplify citations but raise the chance of irrelevant context. Rich metadata improves governance but requires clean source systems and process discipline.

A leader should ask what failure would be most costly. In legal, medical, financial, and security settings, retrieving the wrong version or leaking restricted content may matter more than a slightly slower answer. In customer support, speed and coverage may matter more, but stale troubleshooting steps can still create real harm.

Practitioners should make these choices measurable. Start with a small evaluation set of real questions. Record the source passages that should be retrieved. Compare chunk sizes, overlap, keyword search, vector search, and hybrid search. Keep the setup that retrieves enough evidence with the least noise and the fewest governance exceptions.

The best systems make retrieval inspectable. A user, auditor, or engineer should be able to see which chunk supported an answer, which document it came from, and why it was eligible to be shown.

What sources are authoritative, and which should never be used for answers?
How fresh must the index be: minutes, hours, days, or release cycles?
Which users are allowed to retrieve which chunks?
What evidence must be stored for audits, citations, or incident review?
What quality target matters most: coverage, exactness, speed, cost, or safety?

✅

A practical rule. Choose the simplest chunking and indexing design that retrieves the right evidence for real questions under real permissions.

Key takeaways

Chunking cuts source material into retrievable units; indexing organizes those units so they can be found quickly.
Good chunks preserve meaning, not just a fixed number of tokens.
Strong retrieval often combines keyword search, vector search, and metadata filters.
Chunk size, overlap, and source structure directly affect recall, precision, and ranking.
Indexes must be refreshed, governed, and tested as source material changes.
Real evaluation questions are more reliable than demos when choosing a chunking and indexing design.

Glossary

Chunk: A piece of source content treated as one unit for retrieval.
Index: A data structure that organizes chunks so a system can find likely matches quickly.
Token: A small unit of text, such as a word, part of a word, or punctuation, used by language models and search tools.
Embedding: A numerical representation of text that lets a system compare passages by meaning.
Metadata: Descriptive labels attached to a chunk, such as source, date, author, version, or access rules.
Recall: The degree to which a retrieval system finds the relevant material that exists.
Precision: The degree to which retrieved results are actually relevant rather than noise.
Vector search: A search method that compares embeddings to find text with similar meaning.
Retrieval-augmented generation: A method where a language model writes an answer using source material retrieved from an external collection.