Practical guide · Context engineering
Long context or retrieval: how to choose
Putting a whole corpus in the prompt and retrieving a few passages are two ways to give a model the same knowledge. They fail in different places, cost different amounts, and need different tests. Choose with measurements, not with a preference for one approach.
Published October 19, 2026 · Reviewed October 19, 2026 · 11 minute read
The short answer
Start from the smallest context that can answer the question. When the entire knowledge base fits in the model window, changes slowly, and can be shared with every user, pass it in the prompt and use prompt caching to reduce the repeated cost. When the corpus is larger than the window, changes often, or needs per-user access control, retrieve the relevant passages and send only those. Then run both on the same fixed set of questions and compare quality, token cost, and latency before you commit.
The two approaches fail in different places
Long-context prompting sends the source material with the question and lets the model decide what matters. There is no index, no embedding step, and no top-k cutoff that can drop the one passage a user needs. The cost is that every call carries the full text, and the model does not weigh every token equally. Anthropic describes the model's working capacity as an attention budget that each token spends a little of, which is why adding more text can make an answer worse rather than better.
Retrieval sends a query to a search index, takes back a small set of passages, and puts those in the prompt. Per-call tokens stay low and access control is easier because you only fetch what the user may read. The weak point is now the retriever: if the index is stale, the embeddings miss a paraphrase, or the reranking drops the right passage, the model never sees it. A retrieval pipeline also adds an index, an embedding model, a search step, and their own evaluation surface.
What the published comparisons show
A 2024 study benchmarked retrieval and long context across several public datasets with three models. The authors report that, when resourced sufficiently, long context consistently outperforms retrieval on average, while retrieval keeps a much lower cost. They also propose a router that sends each query to whichever path the model's own reflection picks, and it keeps performance close to long context while cutting computation. The finding is a trade between quality and cost, tested on that benchmark set, not a rule that one approach wins everywhere. The paper is Retrieval Augmented Generation or Long-Context LLMs?.
Position inside the context matters more than length alone. The Lost in the Middle study found that models use information near the start or the end of the input far more reliably than information buried in the middle, even for models built for long inputs. When retrieval puts the best passage first, it does more than shorten the prompt. It also changes where the answer sits.
Adding more text does not guarantee a better answer. Chroma's context rot report evaluated 18 models, including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3, and found that performance grew less stable as input length increased, even on simple tasks. A model can hold a million tokens and still answer less accurately than it does with a few thousand well-chosen ones.
A decision table
| Situation | Lean long context | Lean retrieval |
|---|---|---|
| Corpus size | Fits inside the window with room to spare | Larger than the window, or close to the edge |
| Change rate | Content is stable across many calls | Content changes often, or is updated per user |
| Access control | Everyone who uses the app may read the whole corpus | Different users may read different parts |
| Question shape | Needs synthesis across most of the corpus | Needs a few passages that answer one question |
| Cost profile | Few calls with a large cached prefix | Many calls, each with a small prompt |
Worked example: one support bot, two corpora
Suppose a product team is building a support assistant. One corpus is the public help center, roughly 120 short articles that change a few times a month. The other is the internal engineering wiki, thousands of pages across teams, with runbooks that only some staff may read.
For the help center, the whole corpus fits in the window, and every user may read all of it. Sending it as a cached prefix is simple and avoids a retriever that could miss the right article. For the wiki, the corpus is too large, changes weekly, and carries per-team permissions. Retrieval is the only safe option because the access check happens where the passages are fetched. Anthropic makes a similar size split in its contextual retrieval guidance: below about 200,000 tokens, close to 500 pages, a full prompt with caching can replace a retrieval pipeline.
Most teams end up with a hybrid
The two approaches are not exclusive. A common pattern is to keep a small set of always-relevant material in the prompt and let the model pull more on demand. Anthropic's context engineering post describes this just-in-time pattern, where instructions stay up front and tools such as file search or grep fetch details only when a step needs them. It calls the same idea progressive disclosure: the model discovers relevant context through exploration instead of receiving everything at once.
A second hybrid routes per query. The Self-Route method from the comparison study asks the model to judge whether it already knows enough to answer, and sends only the queries that need it to retrieval. That keeps the cost of retrieval for the easy questions and reserves the full context for the hard ones. Managed file-search tools formalize the retrieval half, as in OpenAI's file search guide, which builds a vector store for you.
How to test the choice
- Write at least 30 real questions with a known correct answer and the passage that supports it. Keep them fixed so both pipelines see the same set.
- Score an answer as correct only when it matches the passage, not when it merely sounds plausible.
- Record input and output tokens, latency, and cost per question for each approach, including the cached and uncached rates.
- Move the supporting passage to the start, middle, and end of a long prompt, and check whether quality changes with position.
- For retrieval, log which passages were returned and whether the supporting passage was in the set. Separate a retrieval miss from a generation miss.
- Repeat the run after a model or index change, because the best choice can move as the model improves.
Prompt caching changes the cost side of this comparison. Anthropic reported that caching a stable prefix cut latency by more than half and cost by up to 90% in its early testing, which is what makes a repeated long prompt affordable. If your prefix changes on every call, the cache does not help and retrieval becomes more attractive.
Limits and common mistakes
- Do not decide from a benchmark average. The comparison study measured specific datasets, so test your own questions.
- A large context window is not a guarantee of accurate recall. Where a passage sits, and how long the prompt is, still matter.
- A retrieval score is not an answer-quality score. Measure both the retriever and the final response.
- Putting a whole private corpus in a prompt is not access control. If users have different permissions, retrieve instead.
- A stale index silently serves old answers. Track when the index was last built and what changed since.
- Prompt caching depends on a stable prefix. Editing instructions or data on every call removes the saving.
- One approach is not permanently better. Recheck the choice when the model, the corpus, or the cost changes.
Decision checklist
- □ The corpus is measured in tokens, not pages.
- □ Access control is defined per user or team, not assumed.
- □ A fixed question set with known answers exists.
- □ Both approaches were scored on the same set, not just the preferred one.
- □ Position and length effects were tested, not assumed away.
- □ Token cost, latency, and cache behavior were recorded.
- □ Retrieval misses are separated from generation misses.
- □ The choice has an owner and a review date.