On retrieval, part 3 of 3

The Data Between the Data

A great deal of what an organization knows lives in the relationships between things, not in any one document. Retrieval that only finds passages never sees it.

A map before you read

One line per section, written by an AI model running entirely in your browser. Free, private, nothing leaves your device.

GraphRAG has had a strange couple of years. It arrived with a lot of excitement, a way to build a knowledge graph out of a pile of documents and retrieve from the graph instead of the pile, and by 2025 the research was turning on it. A benchmark paper revised in February 2026 opens with the problem plainly: “recent studies report that GraphRAG frequently underperforms vanilla RAG on many real-world tasks” (Xiang et al.).

So why build graphs at all? The same paper shows exactly where they earn it. On questions that require connecting facts, graphs win, and by a lot. On questions that need one fact from one place, they don’t, and the cost of building them is real. That split is more useful than either the hype or the backlash, because it points at what graphs are actually for. A great deal of what an organization knows lives in the relationships between things, not in any single document, and retrieval that only finds passages never sees it.

This is the last of three essays on retrieval. The first covered what RAG is and why similarity isn’t relevance. The second covered hybrid search and reranking, the two fixes for finding and ordering the right passages. This one is about what no passage contains.

What Chunking Throws Away

Every retrieval system in the first two essays works on passages. It splits documents into chunks, finds the chunks that best match a question, and hands them to the model. That’s the right design for a question whose answer sits inside one chunk. Plenty of questions don’t work that way.

Take the smart-district platform I worked on, the one that started at the Honda Center. An event happens at a venue. The venue sits in a district. The district’s concession stands sell particular items, and each sale is tied to an event, a stand and a time. Now ask which items sold best at sold-out events in a given district last season. Where is that written down? Nowhere. The answer exists only in the connections between events, venues, districts, stands and sales, and it has to be assembled by following them. Researchers call these multi-hop questions, and they’re where passage retrieval breaks down. The passage that holds the answer usually isn’t the one the question names, so similarity search never finds it.

That’s what I mean by the data between the data. The records are the data. How they connect is data too, often the most valuable kind, and chunking discards it on the way in.

Text Hides Its Relationships

The obvious objection is that documents already describe these relationships. A post-event report says the concert sold out. A concessions report says the stands ran out of hot dogs by halftime. Read them together and the connection is plain. Isn’t it all there in the text?

It’s there for a person. Notice how much of that connection was never written down. Neither report says the stands that ran out were at the venue hosting that concert, or that the concert was the reason they ran out. A reader supplies those links without noticing, from the order of the sentences, from knowing how stadiums work, from having been to one. Linguists call one version of this bridging, where something is understood through an earlier thing it was never explicitly tied to. In their example, “There is a house. The door is red,” nobody wonders whose door it is.

Language models are much worse at supplying those links than they look. In GUMBridge, a corpus of these references published by Georgetown researchers in December 2025, human annotators identified what a bridging reference pointed back to 84% of the time. GPT-5 managed 49%, and the authors concluded that “bridging related tasks remain difficult even with the use of SoTA LLMs” (Levine and Zeldes). An April 2026 study of implicit information found the same gap from another direction. When people and models were asked to extract the relationships implied by a text, humans agreed with most of what the models found but proposed many relationships the models had missed, and where the text only implied which event came first, people inferred the order and models tended to abstain (De Santis et al.).

Chunking makes this worse in a way that’s easy to miss. A person reads the whole report. A retrieval system hands the model a fragment, often without the earlier paragraph that introduced what the fragment refers to. The text looked relational because the reader was doing the relating. Cut the text into pieces and the relationships that lived in the reader’s head don’t come along.

Why Similarity Can’t Follow a Chain

There’s a second, more mechanical reason passage retrieval fails on these questions, and an August 2026 paper measured it directly. Most retrieval pipelines score each chunk against the question on its own, which quietly assumes that a single chunk is enough to answer it. For multi-hop questions, by design, none is, and the paragraph that holds the answer is the one the question doesn’t name (Kumar).

Checking whether a chunk supports the answer worked almost perfectly on single-hop questions, with an AUC of 0.951. On three multi-hop benchmarks it fell to between 0.52 and 0.64, barely better than a coin flip at 0.5. Filtering chunks this way made end-to-end answers worse than not filtering at all, in every configuration tested. What fixed it was breaking the question into its hops and checking each chunk against the step it actually belongs to. Using MuSiQue’s own step-by-step breakdown of its questions, checks on a later hop rose from chance, 0.546, to 0.840. That’s the argument from my essay on finding the seams, showing up inside retrieval: a question that looks like one question is several, and each part needs its own answer.

Models have a related limit with facts they hold in their weights. A 2025 study from researchers at Apollo Research and the UK AI Security Institute found that models trained on two separate facts largely failed to combine them into a two-step answer without reasoning out loud, but succeeded when the facts appeared together in training or in the prompt (Balesni, Korbak and Evans). In other words, models combine facts far more reliably when the facts are in front of them together.

What a Graph Adds

So what does a graph actually add? The first thing is that the relationship gets written down. An edge saying this sale happened at this event, at this stand, in this district doesn’t depend on anyone inferring it from the order of two sentences, so the link that people supply without thinking, and models miss about half the time, is stored as data.

That also changes how retrieval moves. In a multi-hop question, the entity in the middle, like the venue that connects the concert to the stand, is usually the one nobody mentioned. In a graph it’s a node one edge away from something the question did mention, so retrieval can walk to it instead of hoping it resembles the question. HippoRAG 2, which I come back to below, works exactly this way: it starts from the concepts in the question and spreads probability outward through the graph, so a passage can rank highly because it’s connected to what was asked even if it shares none of its words. And when retrieval returns a connected piece of a graph instead of a handful of unrelated chunks, the facts that need combining arrive in the prompt together, which is the condition under which the Apollo study found models could combine them.

The last thing a graph adds is the one people underrate: it can produce facts no record holds. “The best-selling items at sold-out events in this district” isn’t stored anywhere. It’s computed by following sales to events, events to venues and venues to districts, then aggregating. That’s a join, and no amount of passage retrieval performs one.

What the Research Says

The clearest recent evidence comes from two systematic comparisons. In the first, Han and colleagues benchmarked standard RAG against a range of graph-based methods and concluded that the two “exhibit complementary behaviors rather than a consistent winner.” Standard RAG did better on single-hop, detail-oriented questions. On Natural Questions, the best standard RAG scored 64.8 F1 against 63.0 for the best graph method. Graphs did better on multi-hop, reasoning-heavy questions, rising from 60.0 to 63.0 on HotpotQA and from 67.0% to 70.3% on MultiHop-RAG (Han et al.).

The second, GraphRAG-Bench, made the split sharper by separating tasks by difficulty. On simple fact retrieval over a collection of novels, standard RAG with a reranker scored 60.9 and the best graph method, HippoRAG 2, scored 60.1. On complex reasoning over the same text, reranked RAG scored 42.9 and HippoRAG 2 scored 53.4, a gain of more than ten points (Xiang et al.). On medical guidelines the pattern held, with a smaller reasoning gap: 58.6 for reranked RAG against 62.0 for the graph.

Accuracy on the novels collection of GraphRAG-Bench. Source: Xiang et al., “When to use Graphs in RAG,” revised February 2026.

Show as a table

Task	RAG with reranking	HippoRAG 2
Fact retrieval	60.9	60.1
Complex reasoning	42.9	53.4

Notice that the graph doesn’t win by finding better passages. It wins on exactly the questions where no single passage is enough, which is the whole argument for modeling relationships explicitly.

Graphs as Memory

The method that keeps winning those comparisons is worth understanding, because it reframes what a knowledge graph is for. HippoRAG 2, presented at ICML 2025, builds a graph of the entities and facts in a collection, links each one back to the passages it came from, and at query time starts from the concepts in the question and walks outward through the graph to find what’s connected to them (Gutiérrez et al.).

Its authors describe it in terms of human memory: the language model as the neocortex, the graph as a hippocampus that indexes and associates.

The results match the metaphor. Earlier graph methods had improved on associative questions at the cost of basic factual recall. HippoRAG 2 improved associative tasks by about seven points over a state-of-the-art embedding model while holding its own on simple facts. And because new documents become new nodes and edges, it can keep learning without retraining anything, which the authors call non-parametric continual learning.

That connects back to where this series started. The 2020 paper that named RAG described it as combining a model’s parametric memory, its weights, with a non-parametric one it can look things up in. A knowledge graph is a better non-parametric memory than a pile of passages, because it remembers how things relate, not just what was said.

The Cost of Building the Graph

The honest problem is that most of those graphs have to be built, and building them is expensive. Most GraphRAG systems use a language model to read every document and extract entities and relationships, and the quality of everything downstream depends on how well it does. Han and colleagues found that GraphRAG’s results depend heavily on the quality of the graph it constructs.

The serving cost varies enormously too. GraphRAG-Bench measured the average prompt size each method sent to the model. Standard RAG sent about 900 tokens per question and HippoRAG 2 about 1,000. Microsoft’s original GraphRAG, in its local mode, sent about 39,000, and in its global mode the paper’s table reports over 330,000. The authors put it mildly: “GraphRAG’s structured pipeline incurs non-trivial token overhead.”

Is ten points of accuracy worth that? Sometimes. A graph that costs a hundred times more per question has to be worth a great deal more than ten points.

The research also points to a cheaper middle path. Han and colleagues tested running both kinds of retrieval and combining what they found, and on multi-hop questions with a 70-billion-parameter model, that improved on the best single method by 6.4%. Routing each question to one method or the other, the cheaper option, improved it by 1.1%.

When the Relationships Already Exist

Here’s the part most GraphRAG discussions skip. A lot of organizations don’t need to extract a graph from text, because they already have one. It’s in their databases, their foreign keys, their schemas and their APIs. Somebody already decided that an event belongs to a venue and a sale belongs to an event. That’s a knowledge graph, built by people who understood the domain, and it’s more reliable than anything a model will infer from documents.

That’s what we leaned on in the BI agent. The district’s data sat behind a federated GraphQL service spanning the warehouse and the application data, and GraphQL is, quite literally, a graph you can query. Instead of retrieving passages and hoping the relationships came along, we gave the model a tool to query the graph, and skills that documented how the entities joined, which filters were allowed and where the traps were. When a question needed events, venues and sales together, the model traversed relationships that already existed rather than guessing at them from text. I covered how that took us from 70% to 96% in Plain Questions, Valid GraphQL.

So the decision isn’t really “vector RAG or GraphRAG.” It’s where the relationships in your domain live. If they’re buried in prose, in contracts, case notes or research papers, extracting a graph can recover them, at a real cost that has to be measured against the questions you actually get. If they already live in your systems of record, the cheapest and most reliable graph is the one you have. Give the model a way to query it.

Takeaways

If I were building retrieval over an organization’s knowledge today, I’d start by sorting the questions people actually ask into the ones a single passage can answer and the ones that need connections. I’d use the hybrid search and reranking from the second essay for the first kind. For the second kind, I’d look for relationships that already exist in structured systems before paying a model to extract new ones, and I’d expose them as tools the model can query. Where the relationships only live in text, I’d build a graph for that part of the collection, measure its cost per question against the gain, and route only the questions that need it.

A document tells a model what someone wrote. The relationships tell it how the world fits together.