What RAG Is, and Where It Helps

Every time a model ships with a bigger context window, someone declares retrieval dead. If a model can read a million tokens, or ten million, why build an index at all? Just hand it everything.

In July 2025, Chroma published a technical report that tested that idea directly. Its researchers held each task’s difficulty constant and varied only how much text surrounded it, across 18 frontier models from Anthropic, OpenAI, Google and Alibaba. Their conclusion was blunt: “Across all experiments, model performance consistently degrades with increasing input length” (Hong, Troynikov and Huber). They called it context rot, and it means a model that accepts a million tokens doesn’t necessarily make good use of them.

That’s the first reason retrieval is still worth understanding. The second is that most people describing it are describing one narrow version of it. This is the first of three essays on retrieval for AI systems. This one covers what RAG actually is, where it helps, and why the version most teams build first runs into a ceiling.

The Model’s Two Memories

The term comes from a 2020 paper out of Facebook AI Research, which described retrieval-augmented generation as models that “combine pre-trained parametric and non-parametric memory for language generation” (Lewis et al.). That definition is worth slowing down on, because it’s more useful than the way the term gets used now.

Parametric memory is what a model learned in training, stored in its weights. It’s broad, fast and frozen. Non-parametric memory is everything outside the weights that the model can look up while it works: a search index, a database, a set of documents, an API. RAG is any system where the model consults the second memory before it answers from the first.

Notice what that definition doesn’t say. It doesn’t mention embeddings, vector databases or chunking. Those are one way to build the second memory, and the most popular one, but a SQL query is retrieval, a keyword search is retrieval, and a tool call that fetches a customer record is retrieval. When someone says RAG has failed, it’s worth asking which retrieval they built.

Where It Helps

Retrieval earns its place wherever the model’s first memory can’t. The most common case is private data. No model was trained on your contracts, your support tickets or your claims history, and no amount of model quality changes that. The second is fresh data. Weights are frozen at a training cutoff, and the world isn’t. The third is provenance. When an answer is built from retrieved passages, the system can point to them, which matters for any decision someone has to defend.

The fourth is less discussed and, in my experience, the one that decides whether a system can ship at all: permissions. A model’s weights can’t forget what a particular user isn’t allowed to see. A retrieval layer can filter by access before anything reaches the model. In the BI agent I described in Plain Questions, Valid GraphQL, every lookup ran behind an access-control proxy, so a question could only ever reach data the person asking was allowed to see.

Why Bigger Windows Didn’t Kill It

The long-context argument isn’t wrong so much as incomplete. For a small, stable body of text, putting all of it in the prompt is often simpler and better. The question is what happens as the text grows, and the research of the last two years has been consistent about the answer.

Context rot is one part of it. The other is that neither approach wins outright. In LaRA, presented at ICML 2025, researchers built 2,326 test cases across four kinds of questions and three kinds of long documents, tested 11 models at 32,000 and 128,000 tokens, and concluded there was “no silver bullet.” Whether long context or retrieval did better depended on the model, the length of the input, the task and the quality of retrieval, so the answer depends on the system you’re building rather than on either technique.

There’s also the bill. Every token in a context window is paid for on every call, and read again on every call. A system that answers a thousand questions a day against the same million tokens pays to process those million tokens a thousand times. Part of what retrieval does is decide which of those tokens are worth paying for.

Similar Isn’t Relevant

Most RAG systems start the same way. Split the documents into chunks, turn each chunk into an embedding, a vector that places it in a space where similar meanings sit close together, and at query time retrieve the chunks whose vectors sit closest to the question’s. It’s easy to build, and it works well for questions that look like their answers.

The trouble is that a lot of real questions don’t look like their answers. In BRIGHT, presented at ICLR 2025, researchers collected 1,384 real queries from fields like economics, psychology, mathematics and coding, where finding the right document takes reasoning rather than word overlap. The embedding model leading the MTEB leaderboard at the time scored 59.0 nDCG@10 on standard retrieval benchmarks. On BRIGHT, the same model scored 18.3.

The same paper found that having a model reason about the query before retrieving improved results by up to 12.2 points, which tells you where the problem is. The embedding was measuring how much the question resembled each document, not whether the document answered it.

We hit the same wall building the BI agent. Our second version embedded every GraphQL query and retrieved the ones closest to the user’s question, and it topped out at 70%. Embeddings could find a query that looked like the question. They couldn’t tell the model what that query meant or when to use it. Similarity is a reasonable first guess at relevance. It isn’t relevance.

Retrieval Is a Design Decision

So what is retrieval for, if not to work around a small context window? It’s a decision about what a model gets to see, made before the model sees anything. Every choice in it, what to index, how to split it, how to search it and how to order what comes back, shapes the answer as much as the model does.

The next two essays take that decision apart. Retrieve, Then Rerank covers the two fixes for similarity’s blind spots: searching by keyword and by meaning at the same time, and then spending real compute deciding which results matter. The Data Between the Data covers what chunking throws away entirely, the relationships between things, and why so much of what an organization knows lives there.

Retrieval was never a workaround for small context windows. It’s how a model gets to know what it was never taught.

Sources

1. Hong, Troynikov and Huber, “Context Rot: How Increasing Input Tokens Impacts LLM Performance,” Chroma Technical Report, July 2025
2. Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” NeurIPS 2020
3. Li et al., “LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs, No Silver Bullet for LC or RAG Routing,” ICML 2025
4. Su et al., “BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval,” ICLR 2025