Why Basic RAG Fails at Multi-Hop Reasoning  🔗 **Source:** [Why Basic RAG Fails at Multi-Hop Reasoning](https://thenewstack.io/graphrag-multi-hop-reasoning-python/) > [!note]+ Document Notes > > [!info] Provenance > > **Analysis generated:** 2026-08-27 by Claude (Opus 5) via the `readwise-deep-read` pipeline. > > **Source:** The New Stack — https://thenewstack.io/graphrag-multi-hop-reasoning-python/ > > **Published:** 2026-08-27 · **Length:** 1,007 words · 8 headings · article (technical tutorial) > > **Byline:** **Not present in the captured markup.** Reader's `author` field is the bare domain. No individual author is asserted. > > **Register key:** `[STATED]` author-stated · `[EVIDENCE]` demonstrated · `[INFERENCE]` reviewer inference · `[RECOMMEND]` reviewer recommendation. > > **Coverage:** Full article read. **All four outbound links point to The New Stack's own prior articles** — no link to Neo4j documentation, the `neo4j-graphrag` package, or the GraphRAG literature. `[EVIDENCE]` **The code blocks themselves are largely absent from the extraction** — only the `pip install` line survives — so the four implementation steps are assessed as *described*, not as written. > > **Extraction artifact:** `[EVIDENCE]` A promotional link — *"A Claude Code skill was eating 200,000 tokens before answering a single question"* — is captured mid-paragraph, splitting the Acme/BetaTech example. Not the author's text. > # 🏷️ Title: Why basic RAG fails at multi-hop reasoning (and how GraphRAG fixes it) > ## 🧭 Table of Contents > - [📝 Summary](#-summary) > - [📖 Main Content](#-main-content) > - [🗒️ Notes](#️-notes) > - [📚 References](#-references) > - [🗣️ Paraphrase](#️-paraphrase) > - [📰 Newsletter](#-newsletter) > - [✅ Action Items](#-action-items) > - [🧪 Hypotheses](#-hypotheses) > - [📊 Methodology](#-methodology) > - [❓ Questions](#-questions) > - [🔮 Future Research](#-future-research) > - [💡 Implications](#-implications) > - [🧠 Learning Styles Assessment](#-learning-styles-assessment) > - [📊 Accessibility Assessment](#-accessibility-assessment) > - [🌍 DEIB Assessment](#-deib-assessment) > - [✅ Logical Validity](#-logical-validity) > - [🛡️ DARVO Analysis](#-darvo-analysis) > - [🎯 Conclusion](#-conclusion) > ### 🔗 References > `[EVIDENCE]` Four links, **all to The New Stack's own prior coverage**. Named but unlinked: > - **Neo4j** and the **`neo4j-graphrag`** package — the framework the entire tutorial depends on > - **Cypher** — Neo4j's query language (see [the spelling problem](#the-spelling-problem-is-not-cosmetic)) > - `VectorCypherRetriever`, `FixedSizeSplitter` — the two classes doing the work > - GPT-4o, as the extraction model > ## 📝 Summary > **The diagnosis, and it is correct.** `[STATED]` *"The standard RAG assumes that semantic similarity implies relevance, which is not necessarily the case."* > **The demonstration is the article's best asset** — two chunks and one question: > > `[STATED]` Chunk 1: *"Acme Corp acquired BetaTech in 2022."* Chunk 2: *"Sarah Connor was appointed CEO of BetaTech in 2023."* > > Query: *"Who leads the company that Acme Corp acquired?"* > `[STATED]` *"chunks with high similarity to 'Acme Corp' and 'leadership' will be found, but Chunk 2 will be overlooked because 'Sarah Connor' and 'CEO of BetaTech' have nothing to do with 'Acme Corp' semantically."* > `[INFERENCE]` **Two sentences, and the failure is undeniable.** The answer is present in the corpus, distributed across chunks that share no vocabulary with the query, and no amount of better embedding recovers it — because the missing thing is a *relationship*, not a similarity. > **The proposal.** Use an LLM at ingestion to extract entities and relationships into a Neo4j graph; at query time, find an entry node by vector search, then traverse relationships to assemble a subgraph. > **The honest part, and the reason to read it.** `[STATED]` Three trade-offs: ingestion cost is high (an LLM call per chunk), schema design is mandatory (*"the moment you do not provide a proper schema, your LLM will hallucinate entity labels, duplicating nodes of 'Acme Corp', 'Acme Corporation,' and 'Ame'"*), and — the underrated one — `[STATED]` *"Observability is superior: To debug your naive RAG, you need to examine big floating-point arrays. To debug GraphRAG, you only need to look at your Neo4j dashboard to view the nodes."* > **Bottom line.** `[INFERENCE]` A clear, correct explanation of a real failure mode, with **one substantive gap**: the title promises multi-hop and **the implementation described performs a single hop**, working on the example only because the entry point happens to be the middle node. > ## 📖 Main Content > ### The title promises multi-hop; the implementation does one hop > `[STATED]` Step 4: *"We will embed the query, identify the semantic entry point, and finally perform a Cipher query on all the **1-hop connected neighbors**."* > `[INFERENCE]` **Trace the worked example against that and something becomes visible.** The graph is: > ``` > (Acme Corp) -[ACQUIRED]-> (BetaTech) <-[LEADS]- (Sarah Connor) > ``` > The query is *"Who leads the company that Acme Corp acquired?"* — which is **two hops from Acme Corp**: Acme → BetaTech → Sarah Connor. `[EVIDENCE]` But the retrieval enters at the **semantic entry point**, and the article reports the retrieved context as *"BetaTech, its acquisition by Acme Corp, and the fact that Sarah Connor leads it."* > `[INFERENCE]` **The entry point is BetaTech**, and from BetaTech both facts are one hop away. **The example is solvable in one hop only because the entry node sits in the middle of the chain.** Move the query one step further out — *"Who leads the companies acquired by the firm that acquired BetaTech?"* — and one hop fails exactly as vector search did. > `[RECOMMEND]` **Traversal depth is a parameter you have to choose, and the article never says so.** Anyone implementing this should treat depth as a design decision with a cost curve: each additional hop expands the subgraph combinatorially, which is why 1-hop is a sensible default and why the article is right to use it. **What is missing is naming it as a limit**, so the reader knows what to change when a genuinely three-entity question fails. > `[INFERENCE]` This is not a fatal objection — the technique is real and the entry-point-plus-neighbourhood pattern solves a large class of questions vector search cannot. It is a **gap between the title's promise and the implementation's reach**, and it is the kind of thing that surfaces in production rather than in a tutorial. > ### Entity resolution is the hard part and gets one sentence > `[STATED]` *"The moment you do not provide a proper schema, your LLM will hallucinate entity labels, duplicating nodes of 'Acme Corp', 'Acme Corporation,' and 'Ame.'"* > `[INFERENCE]` **This is the single most practically important line in the article and it is a warning rather than a solution.** A schema constrains *labels* — that a node is a `Company` rather than an `Organisation` — and does considerably less for *identity*: whether "Acme Corp" appearing in document 40 is the same node as "Acme Corporation" in document 12. > `[INFERENCE]` **Entity resolution is where real GraphRAG deployments fail**, and it fails silently. A duplicated entity does not error — it produces a graph where traversal from one node misses facts attached to its twin, which looks exactly like the original problem the graph was built to solve. The article's own example is the illustration: if the acquisition attaches to "Acme Corp" and a later filing attaches to "Acme Corporation," the multi-hop query breaks and the failure is invisible in the query result. > `[RECOMMEND]` Anyone building this needs a **deduplication and canonicalisation pass** after ingestion, and the article does not mention that one exists. `[INFERENCE]` **A schema is necessary and not sufficient**, and treating it as sufficient is the most likely way to follow this tutorial correctly and still get a broken graph. > ### The observability argument is the best reason to adopt, and it is buried in a bullet > `[STATED]` *"To debug your naive RAG, you need to examine big floating-point arrays. To debug GraphRAG, you only need to look at your Neo4j dashboard to view the nodes. It is easy to see if the LLM has hallucinated."* > `[INFERENCE]` **This deserves to be the headline rather than the third trade-off.** The stated benefit of GraphRAG is retrieval quality; the *durable* ben Rick Rothenberg for Unsplash+ The current approach to designing LLMs within AI engineering is oversimplified. According to the echo chamber’s view, solving LLM hallucinations is easy: simply design a standard Retrieval-Augmented Generation (RAG) system in which you break your PDFs into 1,000-token chunks, embed them, insert them into a vector database, and perform cosine similarity searches. It works perfectly well…until you actually deploy it. After deployment, companies quickly discover that using only chunked text for retrieval does not work well for complex questions. The standard RAG assumes that semantic similarity implies relevance, which is not necessarily the case. When users ask a “multi-hop” question that requires making connections between Concept A and Concept B through Concept C, standard RAG will not work because these concepts usually don’t coexist in the same chunk of text. RAG also falls apart at global summarization (“what are the major risk factors discussed in all of our compliance reports?”) > “The standard RAG assumes that semantic similarity implies relevance, which is not necessarily the case.” > > If you are designing AI for enterprise systems, you need structured reasoning. Chunking text is fine, but you should stop doing it randomly. What you need is GraphRAG. GraphRAG offers a powerful combination of the structural knowledge of knowledge graphs along with the semantic capabilities of vector search. This tutorial will explain how your current pipeline fails and help you implement a GraphRAG workflow using Python. #### The limits of naive vector search We will now take apart a simple misconception that vector embeddings will save the day! Assume you have a dataset of corporate contracts [stored in a vector](https://thenewstack.io/silent-llm-hallucination-loop/) database. 1. [A Claude Code skill was eating 200,000 tokens before answering a single question](https://thenewstack.io/claude-code-token-reduction/) * **Chunk 1**: “Acme Corp acquired BetaTech in 2022.” * **Chunk 2**: “Sarah Connor was appointed CEO of BetaTech in 2023.” **The Query**: “Who leads the company that Acme Corp acquired?” Naive vector search will embed the query and find the closest chunks. Inevitably, chunks with high similarity to “Acme Corp” and “leadership” will be found, but “Chunk 2” will be overlooked because “Sarah Connor” and “CEO of BetaTech” have nothing to do with “Acme Corp” semantically. The regular RAG approach lacks a structured relationship with different entities. To answer multi-hop queries, you need to understand that there is an `(“Acme Corp”) - [ACQUIRED]` -> `(BetaTech)` relationship, and `(“Sarah Connor”) - [LEADS] -> (BetaTech)` This is where GraphRAG comes in. #### Enter GraphRAG GraphRAG requires the LLM to process the documents during the ingestion step, extracting nodes (entities) and edges (relationships) to create a knowledge graph from your chunks. Unlike traditional approaches, where queries consist of isolated text strings, in GraphRAG we first identify the starting node via vector search and then traverse the graph’s relationships to obtain a highly relevant subgraph for [reasoning with the LLM](https://thenewstack.io/how-diffusion-based-llm-ai-speeds-up-reasoning/). Recently, frameworks such as Neo4j’s neo4j-graphrag have made this architecture accessible to everyone. We’ll use this framework to build a robust pipeline that performs ingestion, schema enforcement, and graph traversal. #### Practical implementation: Building GraphRAG in Python We will build a pipeline [that can ingest unstructured text data](https://thenewstack.io/data-pipelines-serve-ai/), build a knowledge graph, and query the graph via dynamic Cipher traversal. ##### Step 1: Install dependencies You will need an active Neo4j database (local or cloud) and the following packages: | | | --- | | `Bash``pip` `install` `neo4j neo4j-graphrag openai python-dotenv` | ##### Step 2: Initialize connections and production guardrails Production-ready pipelines require proper credential management and resilience. GraphRAG ingestion is extremely expensive in terms of LLM API requests. Without configuring a rate-limit handler, your pipeline will fail with a `RateLimitError` on the first real document. ##### Step 3: Enforcing schema in the knowledge graph builder An unstructured graph is nothing more than an unorganized database. The moment you do not provide a proper schema, your LLM will hallucinate entity labels, duplicating nodes of “Acme Corp”, “Acme Corporation,” and “Ame.” We will build our architecture beforehand and feed it into the pipeline. > “An unstructured graph is nothing more than an unorganized database.” > > *Engineering note*: Under the hood, this pipeline still splits your text using a `FixedSizeSplitter`. The difference is that it uses the LLM to extract the semantic architecture *from* those chunks before saving them. ##### Step 4: The vector Cipher retriever This is where multi-hop reasoning happens. An ordinary vector retriever will provide us with nodes. To travel across the graph, we use a VectorCypherRetriever. We will embed the query, identify the semantic entry point, and finally perform a Cipher query on all the 1-hop connected neighbors. The retrieved context includes BetaTech, its acquisition by Acme Corp, and the fact that Sarah Connor leads it, since we provided a specific schema for retrieving relations. The LLM now has all the facts it needs to provide an accurate response. #### Lessons learned: What engineers must know before adopting GraphRAG If you want to upgrade your RAG system, consider the following trade-offs: 1. **Ingestion cost is high**, whereas RAG costs are low because embedding models are inexpensive. To ingest each document chunk in GraphRAG, you need an LLM like GPT-4o to extract entities from the document. Filter your enterprise data carefully. 2. **Schema design is not optional:** As shown in step 3, you shouldn’t ignore data modeling. The schema is the difference between an efficient retrieval engine and a mess of nodes. 3. **Observability is superior**: To debug your naive RAG, you need to examine big floating-point arrays. To debug GraphRAG, you only need to look at your Neo4j dashboard to view the nodes. It is easy to see if the LLM has hallucinated. Standard vector search is a very powerful solution, but treating it as a one-size-fits-all fix for enterprise AI is an easy way out. Real-world use cases need explicit data modeling and deterministic search capabilities. > “Standard vector search is a very powerful solution, but treating it as a one-size-fits-all fix for enterprise AI is an easy way out.” > > GraphRAG demands more intentionality in its design, strict guidelines to follow, and greater upfront computational effort. However, the rewards are immense: GraphRAG gives you the power to do something truly valuable—multi-hop reasoning over data, which is exactly what enterprises desire., Why Basic RAG Fails at Multi-Hop Reasoning, The New Stack, 01m120t86gs3feawqmenjsjxtj.