Retrieval Evaluation & Grounding — Advanced Retrieval: GraphRAG & Hybrid

A grounded artificial intelligence system is not judged only by how polished its answer sounds. It is judged by whether it found the right source material, used that material accurately, and made its uncertainty visible when the evidence was weak. Retrieval evaluation is the discipline of testing the first part of that chain: did the system bring back the information needed to answer the question? Grounding is the discipline of testing the second part: did the generated answer stay tied to that information instead of inventing details?

The basic problem: answers need evidence

A large language model is good at producing fluent text, but fluency is not the same as truth. If you ask about a company policy, a product manual, a contract clause, or a medical procedure, the useful answer is usually not whatever the model can infer from its general training. The useful answer is the answer supported by the right document, table, record, or knowledge base entry.

Retrieval is the act of finding candidate evidence before the model writes. A user asks a question, the system searches a collection, returns the most relevant passages, and gives those passages to the model as context. The model then writes an answer using that context. This pattern is often called retrieval-augmented generation, or RAG.

Evaluation matters because each step can fail quietly. The search system may miss the best passage. The ranking system may put a weak passage above a strong one. The model may ignore the evidence, overstate it, or cite the wrong source. A good evaluation separates these failures so teams know what to fix.

A RAG system retrieves candidate passages, ranks them, and the model writes an answer grounded in that evidence.

What retrieval evaluation asks

Retrieval evaluation starts with a simple question: when the system searched, did it return the material a knowledgeable human would need? This is different from asking whether the final answer was good. A system can retrieve the right passage and still write a poor answer. It can also write a plausible answer after retrieving the wrong passage, which is more dangerous because the failure is hidden.

The core test set contains questions, the source collection to search, and a human judgment about which passages are relevant. Those judgments are sometimes called relevance labels. They can be exact, such as one paragraph that contains the answer, or graded, such as excellent, useful, weak, and irrelevant.

The most common retrieval measures are built around position. Recall at k asks whether at least one relevant item appeared in the top k results, where k is a small number such as 3, 5, or 10. Precision asks what share of returned items were relevant. Mean reciprocal rank rewards systems that place the first relevant item near the top. Normalized discounted cumulative gain gives more credit when highly relevant items appear early and less credit when they appear late.

Use recall when missing the right evidence is the main risk.
Use precision when irrelevant evidence distracts the model or wastes context space.
Use rank-sensitive metrics when users or models mostly rely on the first few results.
Use graded relevance when some passages are complete answers and others are only supporting hints.

📌

Keep metrics tied to behavior. A retrieval metric is useful only if improving it makes the final system better for the people who use it.

What grounding evaluation asks

Grounding evaluation starts after retrieval. It asks whether the answer is supported by the evidence that was provided. A grounded answer does not need to quote every source word for word, but every factual claim should be traceable to the retrieved material or to a clearly allowed source of knowledge.

There are several distinct grounding checks. Faithfulness asks whether the answer says only what the evidence supports. Answer correctness asks whether the answer satisfies the user question. Citation accuracy asks whether each citation points to the source that actually supports the claim. Completeness asks whether important supported facts were left out. Abstention asks whether the system declined to answer when the evidence was insufficient.

These checks can disagree. An answer can be faithful but incomplete. It can be correct but poorly cited. It can be well written but unsupported. Evaluation should record the specific failure mode, not just a single pass or fail score.

Grounding evaluation asks whether an answer is faithful, correct, well cited, and honest about what it cannot support.

Build an evaluation set before tuning

A useful evaluation set is not just a random pile of questions. It should represent the work the system is expected to do. If the system supports customer service, include common questions, rare edge cases, ambiguous questions, outdated terminology, and questions that should not be answerable from the approved sources. If it supports engineers, include version-specific questions, error messages, and questions that require combining two sources.

Each example should capture the user question, the expected relevant sources, and the expected answer behavior. For retrieval, the label may identify the best passage or document. For grounding, the label may identify the claims that must be supported and the claims that would be unacceptable. If the system must cite sources, the expected citation target should be recorded too.

Teams often begin with a small expert-labeled set, then expand it with production logs after removing personal data and sensitive content. Synthetic questions, generated from documents, can help cover breadth, but they should not replace human review. Synthetic data often mirrors the wording of the source and can make retrieval look easier than it is.

⚠️

Do not tune to the exam. If the same examples drive every design choice, the system may improve on the test while getting worse in real use.

Diagnose the pipeline, not just the answer

When a grounded system fails, the repair depends on where the failure occurred. If the right source was not retrieved, improve chunking, metadata, query rewriting, filtering, or the search method. If the source was retrieved but ranked too low, improve ranking or reduce the number of distractors. If the source was present but the model ignored it, change the prompt, context format, or answer policy. If the answer cited the wrong passage, improve citation extraction and claim-to-source matching.

Chunking deserves special attention. A chunk is a unit of text placed in the search index, often a paragraph or section. Chunks that are too small may lose necessary context. Chunks that are too large may bury the answer and waste the model's context window. Good evaluation exposes this trade-off because retrieval may look strong at the document level while grounding fails at the passage level.

Metadata also changes outcomes. Product version, geography, access level, date, department, and document status can be as important as the text itself. In Microsoft Foundry or any comparable platform, retrieval tests should verify that filters and permissions are honored, especially when identity comes from Microsoft Entra ID or another identity provider.

No relevant result: inspect indexing, chunking, synonyms, and query rewriting.
Relevant result too low: inspect ranking features and reranking behavior.
Relevant result present but unused: inspect prompt instructions and context layout.
Unsupported claim: inspect generation settings, answer policy, and evidence checks.
Wrong citation: inspect claim segmentation and citation selection.

Where a grounded answer fails decides the fix: retrieval and search, or prompting and ranking.

Automated judges help, but they need calibration

Human review is the reference point for evaluation, but it is slow and expensive. Automated judges can speed up iteration by scoring retrieval outputs, comparing generated answers to reference answers, or checking whether claims are supported by cited passages. An automated judge may be a rule, a classifier, a large language model, or a combination.

The danger is that an automated judge can be confidently wrong. It may reward verbose answers, miss subtle contradictions, or treat a plausible statement as supported when the passage does not say it. For this reason, judge quality must be measured against human labels. The practical question is not whether the judge is perfect, but whether it is reliable enough for the decision being made.

A common pattern is to use automated judging for fast regression testing and human review for calibration, launch gates, and disputed cases. Tools in Microsoft Foundry and Foundry Tools can support parts of this workflow, but the important design choice is independent of vendor: keep the judge criteria explicit, sample its mistakes, and watch for drift as documents, prompts, and user behavior change.

⚠️

Do not trust a silent judge. An automated score without examples, disagreement rates, and known blind spots can create false confidence.

The decisions this evaluation supports

Leaders and practitioners use retrieval and grounding evaluation to make concrete trade-offs. One decision is launch readiness: is the system good enough for a limited pilot, a supervised workflow, or direct user access? Another is scope: which topics are safe to answer, and which should route to a human or a traditional workflow? A third is investment: should the team improve content quality, search infrastructure, prompting, model choice, access controls, or monitoring?

The right threshold depends on the cost of failure. A travel policy assistant can usually tolerate a lower risk than a clinical guidance tool or a financial reporting assistant. In high-stakes settings, the system may need strict citation requirements, conservative abstention, human approval, and audit logs. In lower-stakes settings, speed and coverage may matter more, as long as uncertainty is clear.

Good evaluation also creates a shared language across roles. Engineers can see whether the index is improving. Subject matter experts can see whether answers preserve meaning. Product owners can see which use cases are ready. Security and compliance teams can see whether permission boundaries and source restrictions are working.

Set different quality thresholds for different risk levels.
Track retrieval quality separately from answer quality.
Require stronger grounding for regulated, contractual, or safety-critical content.
Use no-answer cases to measure whether the system can refuse gracefully.
Review failures by business impact, not only by metric score.

A practical starting checklist

Start small and make the evaluation inspectable. Choose a representative set of questions, label the sources that should be found, define what a supported answer looks like, and record what counts as a refusal. Run the same set whenever you change indexing, chunking, ranking, prompts, models, or content. Save the retrieved passages, the final answer, the citations, and the scores so failures can be replayed.

Separate offline evaluation from live monitoring. Offline evaluation uses a controlled test set to compare versions before release. Live monitoring watches real traffic for low-confidence answers, missing citations, repeated no-answer patterns, user corrections, and changes in document coverage. Both are needed: offline tests give repeatability, while live monitoring catches surprises.

The goal is not to prove that the system will never be wrong. The goal is to know what kinds of wrong are likely, how often they happen, how serious they are, and which design change reduces them. That is what turns grounding from a vague promise into an engineering practice.

✅

Start with traces. If you store the question, retrieved evidence, answer, citations, and judge result, most failures become diagnosable.

Key takeaways

Retrieval evaluation tests whether the system found the evidence needed to answer.
Grounding evaluation tests whether the answer is supported by the evidence it used.
A single quality score hides important failures such as missing evidence, wrong ranking, unsupported claims, and bad citations.
Evaluation sets should include common tasks, edge cases, sensitive cases, and questions that should not be answered.
Automated judges are useful for speed, but they must be calibrated against human review.
Leaders should set grounding thresholds according to the real cost of being wrong.

Offline evaluation compares versions on a controlled set, while live monitoring watches real traffic for surprises.

Glossary

Retrieval: The process of searching a source collection and returning candidate evidence for a user question.
Grounding: The practice of tying an answer to specific evidence so its factual claims can be checked.
Retrieval-augmented generation: A design pattern in which a system retrieves relevant information and gives it to a model before the model writes an answer.
Relevance label: A human judgment that says whether a document or passage is useful for answering a question.
Recall at k: A retrieval metric that checks whether a relevant result appears within the first k returned items.
Chunk: A unit of content, such as a paragraph or section, that is indexed and retrieved as a search result.
Citation accuracy: The degree to which an answer's cited sources actually support the claims attached to them.
Abstention: The system's decision to decline answering when the available evidence is insufficient or outside policy.