Query Planning & Multi-Hop — Advanced Retrieval: GraphRAG & Hybrid

A simple question can hide a small investigation. If someone asks which supplier caused the most late deliveries after a policy change, the system may need to find the policy date, identify affected suppliers, count deliveries before and after, and compare the results. Query planning is the act of turning that hidden investigation into explicit steps. Multi-hop answering is what happens when one step depends on the result of another.

A query is not always a single search

At first glance, a query looks like a sentence someone types into a search box or a chat interface. The natural assumption is that the system should look for matching text and return the best result. That works when the question and the answer live close together, such as asking for the phone number on a contact page.

Many real questions are different. They require intermediate facts. The answer may not appear in any one document. It may need to be assembled from a policy, a customer record, a product table, a meeting transcript, and a calculation. In those cases, the first job is not answering. The first job is deciding what must be learned.

Query planning is the process of breaking a question into smaller operations: clarify the goal, choose sources, retrieve evidence, transform or join facts, check whether the evidence is enough, and only then produce an answer. A plan can be written by a developer, generated by a large language model (LLM), or built by a mix of rules and model judgment.

What makes a question multi-hop

A hop is a move from one fact or source to the next. In a one-hop question, the answer can be found directly: What is the warranty period for product X? In a multi-hop question, the system must use one result to find or interpret another: Which products owned by customers in region Y have warranties expiring next quarter?

The key feature is dependency. The second search cannot be framed correctly until the first search has returned something useful. If the system does all searches at once, it may retrieve a pile of related material but still miss the chain of reasoning that connects the material to the answer.

Multi-hop work is common in enterprise systems because facts are distributed. Contracts, tickets, invoices, data warehouse tables, email threads, and knowledge articles are written for different purposes. A person knows how to move among them. A system needs an explicit or learned plan.

A customer question may require finding the customer, then their contract, then the clause that applies.
A compliance question may require finding the rule, then the affected system, then the control evidence.
A sales question may require finding the account segment, then recent activity, then open opportunities.
A technical question may require finding the error, then the software version, then the matching known issue.

⚠️

Similarity is not reasoning. A search result can be textually similar to a question yet fail to contain the intermediate facts needed to answer it.

The basic parts of a plan

A useful plan is smaller than a full program but more precise than a wish. It names the subquestions, the sources to consult, the order of work, and the stopping condition. The stopping condition matters because open-ended planning can waste time, spend money, and collect irrelevant context.

The first part is decomposition: split the original question into parts that can be answered independently or in sequence. The second part is source selection: decide whether the next fact is likely to live in documents, databases, application programming interfaces (APIs), logs, or user clarification. The third part is execution: run the searches, calls, or calculations. The fourth part is synthesis: combine the evidence without inventing missing links.

Plans should preserve provenance, meaning the record of where each fact came from. Provenance lets a system cite sources, debug failures, and apply security rules. Without provenance, a final answer may look confident while hiding a weak or forbidden source.

A query plan decomposes the question, selects sources, executes searches and calls, then synthesizes cited evidence.

Fixed plans, adaptive plans, and hybrids

A fixed plan is designed in advance. For example, a support assistant might always identify the product, retrieve the product manual, search known issues, and then produce a troubleshooting answer. Fixed plans are easier to test, cheaper to run, and safer in regulated workflows because their behavior is predictable.

An adaptive plan is built at runtime. The system looks at the user's question and decides the next step based on what it has already learned. Adaptive planning is better for broad, messy domains where the path is not known in advance. It is also harder to control, because each step may change the shape, cost, and risk of the remaining work.

Most serious systems use hybrids. They give the model freedom inside guardrails. A developer may define allowed tools, maximum hops, required citations, approved data sources, and escalation rules. The model can choose among those options, but it cannot invent new privileges or silently bypass policy.

Planning ranges from fixed and predictable to adaptive and flexible, with guardrailed hybrids in between.

Retrieval is only one operation

Retrieval means finding candidate information. In retrieval-augmented generation (RAG), a system retrieves relevant material and gives it to an LLM so the answer can be grounded in external evidence. This is valuable, but retrieval alone does not solve planning. A multi-hop question may need several retrievals, each shaped by what the previous one found.

Some steps are not retrieval at all. The system may need to filter rows, compare dates, calculate totals, call an API, ask the user a clarifying question, or check whether two names refer to the same entity. A query planner should treat search as one tool among many, not as the whole architecture.

Structured and unstructured sources behave differently. Structured data, such as a database table, has fields and relationships that can be queried precisely. Unstructured data, such as a policy document or transcript, is written in prose and usually needs semantic search or reading comprehension. Multi-hop systems often cross that boundary, which is where many failures begin.

📌

Tool choice matters. The best next step may be a database query, a document search, a calculation, or a clarifying question.

Retrieval is only one operation; planners also filter, join, calculate, and call tools to reach an answer.

How plans fail and how to test them

Plans fail in recognizable ways. The system can decompose the question incorrectly, retrieve the wrong evidence, stop too early, take an unnecessary branch, join facts that do not belong together, or produce an answer that sounds plausible but is not supported. Multi-hop work magnifies errors because a mistake in an early step can poison every later step.

Testing should therefore look at the path, not only the final answer. A final answer can be right for the wrong reason, and a wrong answer can hide a good first step followed by one bad join. Good evaluations record the planned steps, tool calls, source documents, intermediate facts, and final response.

A practical evaluation set should include simple questions, hard multi-hop questions, ambiguous questions, questions with no answer in the available sources, and questions that test permissions. If the system uses Microsoft Entra ID or another identity service to enforce access, tests should confirm that planning does not leak facts from sources the user is not allowed to see.

The decisions leaders and practitioners face

The first decision is whether the problem truly needs multi-hop planning. If most questions are direct and the cost of a wrong answer is low, a simple search or RAG design may be enough. If questions cross systems, require calculations, or affect money, safety, compliance, or customer trust, explicit planning becomes more important.

The second decision is how much freedom to give the planner. More freedom can improve coverage, but it also increases latency, cost, and surprise. A leader should ask where predictability matters more than breadth, and where the organization can tolerate exploration. A practitioner should turn that answer into limits: allowed tools, maximum hops, timeout rules, citation requirements, and human review points.

The third decision is what to observe in production. Measuring only thumbs-up and thumbs-down feedback is too shallow. Teams need traces, cost per answered question, tool error rates, retrieval quality, permission denials, unresolved ambiguity, and the share of answers that cite sufficient evidence. These signals show whether the system is solving the right problem or merely sounding fluent.

Use fixed plans for high-volume workflows with stable steps.
Use adaptive plans when questions vary and the correct path is not known in advance.
Set hard limits on tools, hops, time, and data access before launch.
Require evidence and provenance for answers that influence important decisions.
Escalate to a person when the plan cannot find enough support or when the user intent remains ambiguous.

✅

A good planner is modest. The safest systems know when to answer, when to search again, and when to say they do not have enough evidence.

Key takeaways

Query planning turns a user question into ordered steps before the system tries to answer.
Multi-hop questions require one result to shape the next search, calculation, or tool call.
Retrieval is important, but multi-hop systems also need filtering, joining, calculation, clarification, and verification.
Fixed plans are easier to test; adaptive plans cover more varied questions but need stronger guardrails.
Evaluate the route as well as the final answer, including sources, intermediate facts, permissions, and stopping behavior.
Leaders should balance accuracy, cost, latency, flexibility, and risk before choosing a planning style.

Questions that cross systems or carry high stakes justify explicit planning, while direct ones may not.

Glossary

Query planning: The process of breaking a user question into ordered steps that can gather, transform, and verify the facts needed for an answer.
Multi-hop query: A question that requires moving through two or more dependent facts, sources, or operations before it can be answered.
Hop: One move from a current fact or result to the next fact, source, or operation in a reasoning chain.
Large language model: A machine learning model trained on large amounts of text that can generate and interpret language.
Retrieval-augmented generation: A design pattern where a system retrieves external information and gives it to a language model to help produce a grounded answer.
Application programming interface: A defined way for software systems to request data or actions from each other.
Provenance: The record of where a fact came from and how it was used.
Stopping condition: A rule that tells a system when it has enough information or when it should stop trying.