Embeddings & Vector Search — Building AI Apps: Retrieval

A normal search box is good at finding the words you typed. It is less good at finding what you meant. If you search for “refund policy,” a keyword system may miss a document titled “returns and reimbursements.” Embeddings and vector search address that gap by representing text, images, audio, or other data as lists of numbers that place similar meanings near each other. The idea is simple enough to grasp from first principles, but the engineering details matter because small choices about data, models, distance, indexing, and evaluation can change whether the system feels useful or strangely unreliable.

From meaning to coordinates

An embedding is a numeric representation of something. That “something” might be a sentence, a product, a customer support ticket, an image, a user profile, or a piece of code. The embedding is usually a vector: an ordered list of numbers such as 0.12, -0.48, 1.03, and so on.

The purpose of those numbers is not to be read by people. Their purpose is to let software compare items mathematically. If two pieces of text mean similar things, a good embedding model places their vectors close together. If they mean different things, it places them farther apart.

You can imagine a map where each point is a document. On a paper map, nearby points might mean nearby towns. In an embedding space, nearby points mean nearby meanings. Real embedding spaces often have hundreds or thousands of dimensions, not two, but the intuition is the same: distance carries information.

Why keyword search is not enough

Keyword search compares symbols. It looks for exact words, stems of words, phrases, filters, and sometimes synonyms. This is powerful, fast, and still useful. It is also brittle when people and systems use different words for the same idea.

Vector search compares proximity in embedding space. It can retrieve “reset my password” for a query like “I cannot get into my account,” even if the words do not overlap much. This is why vector search is often called semantic search: it tries to search by meaning.

The trade-off is that semantic similarity is not the same as correctness. A vector search system may return a passage that sounds related but does not answer the question. Good search systems often combine keyword search, vector search, filters, and ranking rules rather than treating any one method as magic.

Keyword search is strong when exact terms, names, identifiers, dates, and codes matter.
Vector search is strong when intent, paraphrase, and fuzzy language matter.
Hybrid search combines both signals and often performs better than either one alone.

⚠️

Similarity is not truth. A nearby vector means “related according to the model,” not “factually correct” or “approved for use.”

Keyword search matches exact symbols while vector search matches meaning in embedding space.

How vector search works

A vector search system has two main phases: preparing the collection and answering a query. During preparation, each item is passed through an embedding model, which produces a vector. The system stores the original item, its metadata, and its vector in a vector index.

At query time, the user’s query is embedded with the same or compatible model. The system then compares the query vector with stored vectors and returns the nearest matches. Those matches may be shown directly, fed into another ranking step, or passed to a large language model for retrieval-augmented generation, often called RAG, which means giving the model retrieved context before it writes an answer.

The comparison uses a distance or similarity measure. Common choices include cosine similarity, which compares direction, dot product, which combines direction and magnitude, and Euclidean distance, which measures straight-line distance. The right choice depends on how the embeddings were trained and normalized.

A query is embedded, compared to stored vectors, and the nearest matches feed a grounded answer.

The index makes it fast

If a system has only a few thousand vectors, it can compare a query vector with every stored vector and sort the results. This exhaustive method is simple and accurate, but it becomes expensive as the collection grows to millions or billions of items.

A vector index is a data structure designed to find close vectors quickly. Many production systems use approximate nearest neighbor search, abbreviated ANN. “Approximate” means the system may not always find the absolute closest vectors, but it can usually find very close ones much faster.

This introduces a practical tuning problem. Higher recall means the system finds more of the truly relevant neighbors, but it may need more memory, more latency, or more compute. Lower latency means users wait less, but the system may skip useful candidates. Engineering vector search is often the art of choosing a tolerable trade-off, then measuring it honestly.

📌

Measure the bargain. Approximate search is useful only when its speed gain is worth the relevance it may give up.

Exhaustive search is exact but slow; approximate search trades some recall for lower latency.

Data preparation decides much of the quality

Embeddings are only as useful as the items you embed. For documents, teams usually split long material into chunks before embedding it. A chunk should be large enough to contain a complete idea and small enough that search results stay focused. Splitting by arbitrary character count often works poorly because it may cut tables, procedures, or arguments in half.

Metadata is just as important as vectors. A search result should respect access control, geography, language, product line, publication date, and document status when those attributes matter. Without metadata filters, vector search can return something semantically relevant but operationally wrong, such as an outdated policy or a document the user is not allowed to see.

Cleaning also matters. Duplicate pages, boilerplate footers, navigation text, and stale records can crowd the nearest-neighbor results. If many items say almost the same thing, the top results may look diverse to the index but repetitive to the user.

Chunk content at natural boundaries such as sections, answers, procedures, or records.
Store metadata needed for permissions, filtering, freshness, and auditing.
Remove or reduce boilerplate, duplicates, and obsolete material before indexing.
Keep a stable identifier that links every vector back to its source item.

What leaders and practitioners must decide

The first decision is the job to be done. A legal discovery tool, a product search engine, a customer support assistant, and a code search tool all have different standards for relevance, latency, explainability, and risk. “Make search better” is too vague to guide design.

The second decision is the operating boundary. Some organizations can send content to an external embedding service; others need embeddings generated inside a controlled environment because of privacy, regulation, intellectual property, or contractual limits. The boundary affects model choice, cost, deployment, monitoring, and incident response.

The third decision is how success will be judged. Clicks are easy to collect but can be misleading. Better measures include whether users reformulate fewer queries, whether support agents resolve cases faster, whether retrieved passages actually contain the answer, and whether restricted content is never exposed. A vector system should have an evaluation set before it has a victory lap.

Common failure modes

One failure mode is embedding mismatch. If documents are embedded with one model and queries with another incompatible model, distances may stop meaning what you think they mean. The system still returns results, but the geometry is broken.

Another failure mode is silent drift. New products, new policies, new slang, and new document formats can change what users ask and what the corpus contains. A system that performed well at launch can decay unless teams re-index, refresh evaluation sets, and monitor query behavior.

A third failure mode is treating vector search as a security layer. It is not. Access control must happen through explicit permissions and filters. If a user is not allowed to see a document, the document should not be returned just because it is semantically close to their query.

Use compatible embedding models for stored items and queries.
Rebuild or update the index when source content changes.
Test with real queries, including misspellings, jargon, and ambiguous wording.
Apply permissions before results are displayed or used by another system.

⚠️

Do not secure by similarity. Vector distance cannot replace permission checks, policy enforcement, or audit logs.

Key takeaways

Embeddings represent content as vectors so software can compare meaning with mathematics.
Vector search finds nearby vectors, which often correspond to semantically similar items.
Keyword search, vector search, filters, and ranking are complementary rather than mutually exclusive.
Approximate nearest neighbor search improves speed at scale but must be tuned and measured.
Data preparation, metadata, permissions, and evaluation often matter as much as the embedding model.
Similarity is not truth, authorization, or business approval.

Vector search decays through model mismatch, silent drift, and being mistaken for a security layer.

Glossary

Embedding: A numeric representation of content designed so similar items have similar numbers.
Vector: An ordered list of numbers that can be compared using mathematical distance or similarity.
Embedding space: The multidimensional coordinate system in which vectors are placed and compared.
Vector search: A search method that retrieves items by finding vectors near the query vector.
Semantic search: Search that tries to match meaning or intent rather than only exact words.
Cosine similarity: A measure that compares the direction of two vectors, often used to judge how similar embeddings are.
Approximate nearest neighbor search: A fast search method that finds very close vectors without always checking every vector exactly.
Retrieval-augmented generation: A pattern in which relevant retrieved content is supplied to a language model before it generates an answer.