RAG vs Long Context: Why Bigger Context Windows Didn't Kill Retrieval

RAG is not dead, and the reason is arithmetic rather than loyalty: a long-context query pays for the whole corpus on every single request, while a retrieval query pays for the few thousand tokens it actually needs. Long context did win real ground — small static corpora, whole-document reasoning, prototypes you want working this afternoon — and this page names where, specifically.

If someone senior has told you to delete the retrieval pipeline and stuff everything into a 1M-token window, the numbers below are the ones to bring to that conversation. They are also the numbers you should check before defending RAG out of habit.

Where these facts come from. Every price, benchmark figure and measurement on this page was read on 22 September 2026 from vendor pricing pages, vendor documentation, or arXiv papers. Nothing here was benchmarked or run by me. These are documentation- and paper-derived claims, not field results. Vendor pricing is volatile and every price below carries the date it was checked. Where the published sources are silent — and on frontier-model latency at a full 1M-token fill, they are silent — this page says so rather than estimating.

One number you may have arrived with is missing on purpose. A widely repeated figure putting the cost gap at roughly 1,250x is traced and taken apart in the cost section below, because it does not survive being checked and this page would rather lose the headline than print it.

Did 1M-token context windows make RAG obsolete?

No. Million-token context windows changed which problems need retrieval, not whether retrieval is needed. The per-token billing model, the quadratic cost of prefill attention, and the gap between advertised and effective context length all survived the jump from 128K to 1M, and each of them independently keeps retrieval in production systems.

That is the short answer. The honest version has two halves, and the first half is a concession.

What actually changed

Long context genuinely won several classes of work when context windows reached a million tokens, and they are not edge cases.

Prototypes. A retrieval pipeline is chunking strategy, an embedding model, a vector store, a reranker, an evaluation harness and a refresh job. Pasting a corpus into a prompt is none of those. For a proof of concept that has to exist by Thursday, long context is simply the correct engineering decision, and teams that build a retrieval pipeline first are usually optimizing something they have not yet shown anyone wants. This matters more where engineering time is the scarce resource relative to API spend — a weighting that differs by market and by team, and one worth making explicit in your own planning rather than assuming the US-benchmark answer.

Whole-document reasoning. Some questions cannot be answered from retrieved chunks because the answer is a property of the document as a whole. “Does this 300-page contract contradict itself?” is not a retrieval query. Neither is “summarise the argument of this book” or “which of these 40 incident reports share a root cause?”. Chunk-based retrieval answers questions whose evidence is local. Long context answers questions whose evidence is distributed, and that is a real class of work it does better.

Small static corpora. If your whole knowledge base is a 60,000-token internal handbook that changes quarterly, a retrieval layer over it is infrastructure you will maintain for no measurable benefit. Put it in the prompt.

Recall of the exact wording. Retrieval can miss. A chunk that should have been returned and was not is a silent failure, and it is the most common reason a RAG system gives a confidently wrong answer. Long context does not have that failure mode, because nothing was filtered out. It has different failure modes, covered below, but this one it genuinely avoids.

The peer-reviewed evidence supports the concession rather than undercutting it. In “The Token Tax of Epistemic Accuracy” (arXiv:2606.20898), long-context prompting reached 73.1% correctness against 65.4% for semantic RAG — evaluated over 972 answers across three machines using two small language models in a manufacturing-safety domain, with expert validation. Scope that carefully, because it is not a frontier-model or 1M-token result. But in that study, the long-context side was the more accurate one.

What didn’t change

Three properties of long-context inference carried straight across the jump from 128K to 1M tokens, and a fourth problem was never the context window’s to solve. Together they are the reason RAG’s obituary is premature.

You pay for every input token, on every request. The context window is not storage. It is a per-request payload that is re-billed each time you send it, whether or not any of it changed since the last query. A 1M-token corpus sent with 10,000 queries is 10 billion input tokens, not one billion-token upload. This is the single fact that most “just use the big window” arguments quietly skip, and the cost section below turns it into arithmetic.

Prefill attention is quadratic in sequence length. The prefill stage — where the model processes your input before emitting the first output token — runs self-attention whose cost scales with the square of the input length. Time to first token therefore grows super-linearly as you add input, which is an architectural property, not an implementation detail somebody will optimize away next quarter. The latency section covers what has and has not been measured about this.

Advertised context length is not effective context length. A model that accepts 1M tokens is not a model that reliably uses 1M tokens. The NoLiMa benchmark (Modarressi et al., arXiv:2502.05167v3, 2025-07-09) tested 13 models that claim at least 128K of context; at 32K input, 11 of them dropped below 50% of their own short-context baseline. The accuracy section takes this apart properly.

And retrieval quality was never the context window’s problem to solve. If your chunking splits a table across two chunks, if your embedding model was trained on a different domain, if your reranker is not there — those defects produce bad answers at 8K of context and bad answers at 1M. Moving to long context removes the retrieval step and therefore removes retrieval bugs, but it replaces them with position and distraction effects rather than eliminating the underlying problem of getting the right evidence in front of the model.

What does a long-context query actually cost compared with RAG?

A long-context query costs roughly one to two orders of magnitude more than a retrieval query per request under production assumptions — and the gap widens with corpus size and narrows with cache hit rate. That range is the honest headline. Anyone quoting you a single multiplier without stating corpus size, tokens retrieved per query, model, price date and cache hit rate is quoting a number that cannot be checked.

The billing model in one sentence

You are billed for every input token in every request, so the cost of a long-context architecture is the size of your context multiplied by your query volume, and the cost of a RAG architecture is the size of your retrieved evidence multiplied by your query volume.

Two consequences follow immediately. First, the ratio between the two is approximately the token ratio — corpus size divided by retrieved tokens — before any price-tier or caching effects. Second, the absolute spend is driven by query volume, which means the architecture that is fine at 1,000 queries a month can be indefensible at 100,000.

Tap to enlarge

Long-context pricing is also not linear across the window on every provider. Google charges $2.00 per million input tokens up to 200K for Gemini 3.1 Pro Preview and $4.00 above 200K (ai.google.dev pricing, checked 2026-09-22). Gemini 2.5 Pro has the same shape at $1.25 and $2.50, though that model is flagged as deprecated on Artificial Analysis as of 2026-09-22 and should not be used as your current reference. Crossing 200K tokens doubles the unit price as well as the volume, so cost rises faster than your token count does — the opposite of the volume discount most people intuitively expect.

A worked example you can recompute

Below is a worked example, not a benchmark. Every assumption is printed so you can substitute your own numbers and get a different, correct answer.

Assumptions

Per-query input cost

That is 67x at a 200,000-token corpus and 667x at a 1,000,000-token corpus. The jump is larger than the 5x growth in corpus size because the >200K price tier doubles the unit price on top of the volume increase. Note what this does not say: it is not a measurement, it is multiplication, and it holds only for the assumptions in the table above.

Tap to enlarge

Monthly input spend at the 1M-token corpus

Run this with your own corpus size and volume before you argue about it with anyone. If your corpus is 40,000 tokens and you serve 500 queries a month, long context costs you $40 and the entire debate is moot — which is exactly the point of printing the arithmetic instead of a ratio.

What the published research reports

The only peer-review-track, head-to-head measurement found during research is “The Token Tax of Epistemic Accuracy” (arXiv:2606.20898, abstract read 2026-09-22), which reports long-context prompting costing 26x more per query than semantic RAG across 972 answers on three machines, using two small language models in a manufacturing-safety domain with expert validation.

Twenty-six is far below the worked example above, and the difference is instructive rather than contradictory. That study used a small corpus and small models; the ratio is approximately the token ratio, so a small corpus produces a small ratio. A measurement at 26x and a calculation at 667x are consistent with each other once you notice they are describing different corpus sizes. Anyone presenting either one as “the” cost gap has dropped the variable that determines the answer.

Two limitations of that paper matter for how much weight to put on it: the PDF body did not extract during research, so the figures cited here come from the abstract and its deeper methodology was not independently inspected; and small language models in one domain do not generalise to frontier models on a 1M-token fill.

The costs the RAG side is usually not charged for

The RAG side of a cost comparison carries real infrastructure — embedding, index storage, reranking and maintenance — and almost every published ratio, including the worked example on this page, sets all of it to zero and compares input tokens alone. That is not honest accounting, and correcting it narrows the gap.

What the RAG side actually costs:

Embedding, once per document. Using text-embedding-3-small at $0.02 per million tokens (OpenAI pricing, checked 2026-09-22), embedding a 10-million-token corpus costs about $0.20 — one time, not per query. Re-embedding on a model change costs the same again. This is genuinely small, and it is small because it is amortised across every query the corpus ever serves; the structural point is that it scales with corpus churn, not with traffic.

Index storage, which is not free. Vector indexes carry the vectors plus the index structures, and for search to be fast that state usually lives on provisioned disk or in memory rather than in cheap object storage. If your retrieval corpus is a second copy of data that already exists in your warehouse, you are paying storage twice, at two different rates. That duplication is avoidable in architectures where the retrieval corpus stays in open table formats instead of a second proprietary copy — though moving the corpus into the lake moves the storage cost rather than deleting it, and the indexes still have to live somewhere.

Reranking and retrieval compute. A cross-encoder reranker over 50 candidates is a model call per query. Hybrid search runs two retrievals. These are per-query costs on the RAG side and they belong in the comparison.

Maintenance, and it is the expensive one. The sync job, the re-embedding pipeline, the chunking strategy someone revisits every quarter, the evaluation harness, the on-call rotation for “retrieval is returning stale documents”. None of this appears on an API bill and all of it appears in a headcount plan. Where engineering time is expensive relative to token spend, this can invert the whole argument for a small corpus — which is the substance behind “just use long context, it’s simpler”, and it is a fair point that deserves a real answer rather than a sneer.

None of this closes a 667x gap. All of it moves a 26x gap meaningfully, and it is the reason a three-figure ratio should make you suspicious rather than triumphant.

Why you should distrust a bare ratio, including this page’s

A RAG-versus-long-context cost ratio is a function of at least eight variables — corpus size, tokens retrieved per query, model, price tier, cache hit rate, cache write cost, RAG-side infrastructure and query volume. Published as a single number with none of them stated, such a ratio is not a finding; it is a number shaped like a finding. The most widely circulated example in this debate is worth auditing in full, because the method generalises to any ratio, including the ones on this page.

The claim. A figure of roughly 1,250x for the RAG-versus-long-context cost gap, described as documented across multiple enterprise benchmarks, circulating across several secondary blogs and attributed variously to Elasticsearch Labs or to a document titled “RAG vs Long Context: Enterprise Production Benchmarks, industry analysis 2026”.

What checking it produced. The Elastic page the attribution points to (elastic.co/search-labs/blog/pre-computed-context-llm-agent-costs) was fetched on 2026-09-22. It contains no 1,250x figure, and it is not a RAG-versus-long-context comparison. It compares baseline RAG against a “Knowledge Indicators” approach on BrowseComp-Plus with Claude Sonnet 4.6 under a 43-step budget, reporting a reduction from 174.8M to 48.3M input tokens — 72%, or about 3.6x. The other trail leads to keepmyprompts.com, which cites the “industry analysis 2026” document with no URL at all. There is no primary source to reach.

That is a citation-laundering chain: one unsourced number, repeated until repetition looks like corroboration.

The audit that produces this result is four questions, and it works on any ratio:

Apply the same four questions to this page. The 667x figure is a multiplication of two published prices by two stated token counts; you can reproduce it on a phone, and you should, because a ratio you cannot reproduce is one you are trusting rather than using. The 26x figure from the Token Tax paper is a real measurement with a real scope, and that scope — two small language models, one domain, small corpus — is the part that would be dropped first if someone quoted it at you.

The same logic condemns a second figure that circulates alongside the first: a “30-60x latency gap”, quoted as RAG at 1-2 seconds against long context at 30-60 seconds. It traces to the same unlinked reference. The direction is defensible from the prefill mechanism, and the latency section below explains why. The specific numbers, as a sourced benchmark for frontier 1M-token APIs, are not.

Does prompt caching close the cost gap?

Prompt caching narrows the cost gap substantially — by roughly an order of magnitude on cached reads — but it does not close it, and it fails hardest in exactly the workloads that made someone consider RAG in the first place. Caching is the strongest argument for long context, and it deserves to be costed rather than waved at.

What caching actually costs

Cached input is cheaper to read and more expensive to write than ordinary input. The write premium is the part that decides whether caching helps you.

Sources: claude.com/pricing, ai.google.dev/gemini-api/docs/pricing, developers.openai.com/api/docs/pricing, all read 2026-09-22.

Three structural facts sit behind that table.

Reads are roughly a tenth of base input; writes cost more than base input. Anthropic’s prompt-caching documentation puts cache reads at about 0.1x base input and cache writes at 1.25x for the 5-minute TTL or 2x for the 1-hour TTL (checked 2026-09-22). The pricing page corroborates those ratios: Opus 5’s $0.50 read is 0.1x its $5.00 input, and its $6.25 write is 1.25x. OpenAI’s 1M-context tier applies a 90% cached-input discount, the same shape; GPT-4.1’s discount is 75%.

Those three ratios are all the documentation states, and the break-even falls out of them by arithmetic rather than by citation. Write once then read: at the 5-minute TTL, two requests cost 1.25x + 0.1x = 1.35x against 2x uncached, so caching pays back on the second request. At the 1-hour TTL the write is 2x, so two requests cost 2.1x against 2x — still behind — and the third request is where it turns, at 2.2x against 3x. Substitute your own TTL and request count; the shape does not change.

Google charges for cache residency by the hour. Gemini context caching adds $4.50 per million tokens per hour of storage on top of the per-token read price (ai.google.dev pricing, checked 2026-09-23). A 1M-token corpus held in cache costs $4.50 an hour whether or not anyone queries it — $108 a day, $3,240 over 30 days, per distinct cached corpus. Spread across 1,000 queries in that hour, storage adds 0.45 cents each. Spread across one query, that query carries $4.50 of storage before a single token is read. Caching is a bet on query density, and the bet is billed while you wait for traffic.

There is a minimum size below which caching is unavailable. Gemini’s documented caching minimums are 2,048 tokens for 2.5 Flash and 2.5 Pro, and 4,096 tokens for Gemini 3.1 Pro Preview and the 3.5-3.8 Flash line (ai.google.dev caching docs, checked 2026-09-22).

Where caching genuinely wins

Prompt caching beats retrieval when one corpus is queried repeatedly and does not change between queries.

A support agent answering questions against one product manual. A code assistant reasoning over one repository for the length of a session. A legal review tool where forty questions get asked about the same contract in the same sitting. In all three, the corpus is written to cache once and read many times, the write premium amortises across the second query onward, and the hourly storage charge is spread across dense traffic.

Work it through on the earlier assumptions. A 1M-token corpus with Claude Opus 5 costs $5.00 per query uncached. Written once at $6.25 and then read at $0.50, a hundred queries against it cost $6.25 + 99 × $0.50 = $55.75, against $500 uncached — a 9x reduction. That is a real saving and it should change your architecture decision if your workload has that shape.

What it does not do is reach RAG’s $0.015 per query on the same model. At a sustained 100% cache hit rate the gap against retrieval is still about 33x on Claude Opus 5, and about 67x on Gemini 3.1 Pro Preview before storage charges. Caching moves long context from two orders of magnitude to one. It does not move it to parity.

Why caching doesn’t close the gap at high corpus cardinality

Prompt caching is a prefix match. The cache is keyed on an exact token prefix, and any byte that changes inside that prefix invalidates everything after it. This mechanism is the reason caching scales with repetition and collapses with variety.

The consequence has a name worth using in a design review: corpus cardinality. If your workload always sends the same corpus, you have one cache entry. If it selects a different customer’s documents per query, you have as many cache entries as customers, each needing its own write at 1.25x and — on Gemini — its own hourly storage charge. Ten distinct 1M-token corpora held for a day is 10 × 24 × $4.50 = $1,080 per day in storage alone, before any query is served.

Tap to enlarge

Three workload properties break caching specifically:

• Many distinct corpora. Per-tenant, per-customer or per-project document sets multiply cache entries linearly. Each entry pays its own write premium and its own residency.
• A corpus that changes. Adding one document to the front of the prefix invalidates the entire cache behind it. Append-at-the-end helps; edit-in-the-middle does not. A corpus updated hourly with a 5-minute TTL is a corpus being rewritten, not cached.
• Sparse traffic per corpus. Ten thousand corpora each queried twice a week never reach the two-to-three-request break-even inside a TTL. Every query pays a write.

There is a real and well-known reason caching still helps more than a uniform-access model would predict: document access in retrieval workloads is skewed, so a minority of documents absorbs a large share of queries and a small cache captures a disproportionate share of traffic. Research quantifying that skew was encountered during this work but not verified against a primary source, so this page states the qualitative point and prints no figures for it. Measure your own access distribution — it is a one-query answer from your retrieval logs and it is the input that decides whether caching is your answer.

The routing rule that falls out: cache when few corpora are queried often; retrieve when many corpora are queried rarely. Most production systems are the second shape, which is why caching narrows the argument rather than ending it.

How much slower is a long-context query?

A long-context query is slower than a retrieval query at the same output length, and the gap grows super-linearly with input size, because prefill attention costs scale with the square of the sequence length. The specific magnitude for a frontier model at a full 1M-token fill has not been published, and this page will not estimate it.

Why TTFT grows with input length

An LLM request has two stages with very different cost curves. Prefill processes the entire input prompt and produces the first output token. Decode generates each subsequent token one at a time. Time to first token is essentially prefill time.

Self-attention in prefill compares every token against every other token, so its cost scales with the square of the input length (VSPrefill, arXiv:2603.04460, §2.1). Doubling the input more than doubles prefill work. This is a property of the attention architecture, not of any particular serving stack, and it is why “we’ll just wait for the hardware to catch up” is a weaker argument than it sounds — the curve is steeper than the hardware improvement.

Retrieval inverts the shape. A RAG pipeline spends time on embedding the query, a vector search, optionally a rerank, and then a prefill over a few thousand tokens instead of a few hundred thousand. Those retrieval steps are real latency and they are not free. They are also roughly constant with respect to corpus size, whereas long-context prefill is not.

What has actually been measured

The clearest published measurement of the prefill curve comes from the VSPrefill paper (arXiv:2603.04460, Guanzhong Chen, 2026-03-03): at 256k tokens of input, self-attention consumed 57.76 seconds, or 89.51% of total time to first token; at 1M tokens, attention time “exceeds one hour”, or 98.56% of latency — measured on Qwen3-4B-Instruct running on H20 GPUs.

Read that scope carefully, because it is the whole reason the figure is usable. Qwen3-4B-Instruct is a small model and H20 is not frontier-class serving hardware. These numbers are not what a frontier API will do, and quoting them as if they were is the same mistake the 1,250x chain made. What they are good for is the shape: attention goes from 89.51% of TTFT to 98.56% of TTFT between 256k and 1M, which tells you the quadratic term dominates well before you reach the advertised window limit.

What is not available: frontier-model TTFT at a full 1M-token fill. No published measurement was found during research on 2026-09-22. Artificial Analysis publishes TTFT for Gemini 2.5 Pro at 24.10 seconds on AI Studio and 30.10 seconds on Vertex, but those are median P50 figures at a 10,000-token workload, not at 1M, they include reasoning time, the model is flagged deprecated, and the page states long-input workloads are no longer being updated. Presenting a 10K-token measurement as evidence about 1M-token behaviour would be dishonest, so it is not used here. If you need the number, measure it on your own workload; if someone quotes it to you, ask which input length it was measured at.

What this means for your p99

The latency question is not really “how many seconds”. It is whether the workload is interactive.

Interactive workloads — a chat interface, an autocomplete, anything where a human is watching a cursor — are governed by TTFT, and TTFT under long context is dominated by prefill over input the user did not ask you to read. This is where retrieval’s roughly constant retrieval cost wins structurally, not by a particular multiple.

Asynchronous workloads — nightly document analysis, batch classification, a report generated while nobody waits — are governed by throughput and cost, not by TTFT. A 40-second prefill in a job that runs at 02:00 is not a problem. If your workload is genuinely async, latency should not be the variable deciding your architecture; cost should be.

The second-order effect worth planning for: prefill is the expensive stage and it is where your concurrency limit lives. A serving tier saturated by long prefills has less capacity for everything else, so a long-context feature added beside an existing interactive one can degrade the interactive one’s tail latency without any change to that feature’s own code. Measure p99 under mixed load, not per feature in isolation.

Does accuracy suffer when you stuff everything into the window?

Accuracy degrades as input length grows, non-uniformly and well before the advertised limit — but long context is not automatically less accurate than retrieval, and at least one peer-review-track study found the opposite. Both halves of that sentence are load-bearing.

Advertised context length vs effective context length

Advertised context length is the input a model will accept. Effective context length is the input over which it still performs. They are not the same number and the difference is large.

The NoLiMa benchmark (Modarressi et al., arXiv:2502.05167v3, 2025-07-09) evaluated 13 models that each claim at least 128K of context. At 32K tokens of input, 11 of the 13 dropped below 50% of their own short-context baseline. GPT-4o went from 99.3% at base length to 69.7% at 32K.

NoLiMa’s Table 3 reports effective context length against claimed length at an 85% performance threshold — that is, the longest input at which a model still retains 85% of its short-context accuracy. GPT-4o claims 128K and has an effective length of 8K on that measure. The threshold is what makes the number meaningful: at a lower bar the effective length is longer, so quote the threshold whenever you quote the figure.

Tap to enlarge

Why needle-in-a-haystack scores look better than reality

Vendor long-context scores usually come from needle-in-a-haystack tests, where a specific sentence is planted in a long document and the model is asked to find it. Those scores are high and they overstate real capability, because the needle usually shares literal wording with the question.

NoLiMa’s design removes that shortcut, and it quantified the difference: ROUGE precision between question and target is 0.069 in NoLiMa against 0.905 in vanilla needle-in-a-haystack. In plain terms, the standard test lets a model succeed by matching words, which is close to keyword search; NoLiMa requires it to make a semantic connection with almost no lexical overlap. The collapse from 99.3% to 69.7% appears when the lexical shortcut is taken away.

Chroma’s Context Rot research (Hong, Troynikov and Huber, 2025-07-14) reaches a compatible conclusion across 18 models including GPT-4.1, Claude 4, Gemini 2.5 and Qwen3: performance “grows increasingly unreliable as input length grows”, and the degradation is non-uniform rather than a smooth curve you can plan around. It argues directly that needle-in-a-haystack “is fundamentally a simple retrieval task” and “may not be representative of flexible, semantically oriented tasks”. Chroma sells a vector database, so that is an interested party’s description of a problem its product solves — treat the framing as interested. The finding itself is consistent with NoLiMa, which has no such commercial stake, and the two together are stronger than either alone.

“Lost in the middle”, and what superseded it

“Lost in the middle” is the finding that model accuracy follows a U-shaped curve over the position of relevant information in a long input: evidence at the beginning or the end is used more reliably than evidence in the middle. It comes from Liu et al., published in TACL in 2024 from a 2023 preprint, and it is the origin of most of what people say about long-context accuracy.

It is real, it is peer-reviewed, and it is the oldest evidence in this discussion. It describes models from before the 1M-token generation. Citing it as the current state of long-context behaviour, as much of the secondary writing on this topic still does, skips two years of better measurement. NoLiMa and Context Rot test what actually breaks on current models, and they should be the citations you reach for. This page treats “lost in the middle” as the origin of the idea rather than as evidence about a model you might deploy today.

The case against RAG on accuracy

Retrieval is not reliably more accurate than long context, and the strongest published evidence on this question runs against the pro-retrieval conclusion. That evidence belongs in the accuracy section rather than in a caveat at the bottom.

In the Token Tax study, long context was more accurate than retrieval: 73.1% correctness against 65.4% for semantic RAG (arXiv:2606.20898, abstract read 2026-09-22; 972 answers, three machines, two small language models, one manufacturing-safety domain, expert-validated). That is a peer-review-track result finding directly against the intuition that retrieval improves accuracy by filtering noise.

The mechanism is not mysterious. Retrieval can fail to return the evidence. When the right chunk is not retrieved, the model answers from what it got, and it answers confidently. Long context does not have that failure mode.

The broader literature does not settle cleanly in either direction, which is itself the finding:

• Li et al. (arXiv:2407.16833, EMNLP 2024 industry track, submitted 2024-07 and revised 2024-10) found long context consistently outperforming RAG when sufficiently resourced, while RAG cost significantly less. That paper predates the 1M-token generation, which matters for how far to carry it.
• A later evaluation (arXiv:2501.01880, submitted 2024-12-27) found long context ahead on Wikipedia-style QA but RAG holding advantages on dialogue-based and general queries — and, importantly, that summarization-based retrieval performed comparably to long context while chunk-based retrieval underperformed both.

That last clause is the most actionable sentence in the accuracy literature. How you retrieve matters as much as whether you retrieve. A team comparing naive fixed-size chunking against long context and concluding that retrieval is inferior has measured their chunking strategy, not the architecture.

So the honest accuracy position: long context avoids retrieval misses and is genuinely better at whole-document reasoning; it degrades non-uniformly as input grows, well before the advertised limit; and a well-built retrieval pipeline outperforms a badly built one by more than either architecture outperforms the other.

When is long context the right choice?

Long context is the right choice when the corpus is small and static, the query volume is low, the task needs the whole document at once, or the workload is asynchronous. Retrieval is the right choice when the corpus is large or changing, the query volume is high, the workload is interactive, or the answer needs citations.

Where long context wins outright

• The corpus fits comfortably and rarely changes. Under roughly 100,000 tokens, a retrieval layer is infrastructure whose maintenance cost exceeds its token saving at most volumes. Compute it rather than assuming: at 60,000 tokens and $2.00/MTok, a query costs $0.12.
• Query volume is low. The per-query gap only becomes real money when multiplied. A few hundred queries a month against a modest corpus is not a cost problem, regardless of the ratio.
• The task is whole-document reasoning. Cross-document contradiction detection, full-document summarisation, “which of these reports share a root cause” — the evidence is distributed, so chunk retrieval is the wrong primitive.
• The workload is asynchronous. Nobody is watching the cursor, so TTFT is not a constraint and cost is the only variable.
• One corpus is queried repeatedly inside a session. Prompt caching amortises the corpus across the session, which is a genuine long-context win rather than a concession.
• You are still proving the idea works. Retrieval infrastructure built before product-market fit is a common and expensive mistake.

Where retrieval still wins

• The corpus is larger than the window, or larger than you want to pay for. Above roughly 200K tokens, Gemini’s price tier doubles; above 1M, the corpus does not fit at all and a selection step is not optional, it is mandatory. At that point the only question is whether your selection step is designed or accidental.
• The corpus changes. Every change invalidates the cache prefix, so a corpus updated during the day is a corpus rewritten to cache during the day.
• Query volume is high. The cost gap is per-query, so it multiplies: at 100,000 queries a month against a 1M-token corpus, the worked example puts long context at $400,000 of input spend against $600 for retrieval.
• The workload is interactive. TTFT is dominated by prefill, and prefill is dominated by input the user did not ask you to read.
• You need citations. Retrieval returns identified chunks with document IDs, which is how a user verifies an answer. A long-context answer can cite, but the citation is generated rather than structural — it is the model’s report of where it looked, not a record of what was fetched. For a regulated or high-stakes workflow that difference is the whole audit trail.
• Corpus cardinality is high. Many distinct corpora, each queried rarely, is the shape that defeats caching and the shape retrieval handles natively.
• Data cannot leave a boundary. If the corpus is subject to residency rules — India’s Digital Personal Data Protection framework being one applicable case — then sending it to a hosted long-context API may not be a decision cost gets to make. A retrieval architecture lets you keep the corpus inside the boundary and send only the retrieved evidence, which is a smaller and more auditable disclosure.

A break-even you can compute

The break-even is not a universal number, and any page that gives you one is guessing at your corpus. It is a short calculation on your own figures.

1. Tokenise your corpus. Call it C.
2. Count tokens in a typical retrieved prompt — top-k chunks plus the query plus instructions. Call it R. Three thousand is a reasonable starting estimate; measure yours.
3. Take your provider’s current input price at the tier C falls into. Remember that Gemini’s price doubles above 200K, so a corpus at 210,000 tokens costs more than twice a corpus at 190,000.
4. Per-query token ratio = C / R. Multiply by the price-tier ratio if C crosses a tier boundary. That is your uncached cost multiple.
5. Apply your measured cache hit rate h: blended cost per query ≈ h × (cached read price × C) + (1 − h) × (cache write price × C). Use your real hit rate from logs. Assuming 90% because it sounds achievable is how these estimates go wrong.
6. Multiply by monthly query volume on both sides, then add the RAG side’s index storage, reranking and maintenance costs.
7. Compare against engineering cost. If the annual difference is smaller than the cost of building and maintaining the retrieval pipeline, long context is the cheaper architecture and the ratio was irrelevant.

Step 7 is the one most comparisons omit, and it is the step that most often decides the answer for a small team.

The architecture most teams should actually build

Route by query, not by doctrine: send queries that need a narrow slice of a large corpus through retrieval, and send queries that need whole-document reasoning or that hit a small hot corpus through long context. That is the recommendation. The condition that would change it: if your entire corpus is small and static and your volume is low, build neither router nor retrieval pipeline — put the corpus in the prompt and revisit when one of those three facts changes.

Route by query, not by doctrine

Hybrid routing is not a compromise reached by splitting the difference. It has a named method in the literature: Self-Route (Li et al., arXiv:2407.16833, EMNLP 2024 industry track), which routes each query between RAG and long context based on the model’s own self-reflection about whether it can answer from the retrieved context. The paper found long context consistently outperforming RAG when sufficiently resourced while RAG cost significantly less, and proposed routing as the way to get most of the quality at a fraction of the cost. It is from 2024 and predates the 1M-token generation, so treat it as the origin of the design pattern rather than as a current benchmark.

What a router looks like in practice, in increasing order of effort:

1. Route on corpus size. If the selected corpus is under your threshold, send it whole; otherwise retrieve. This is a conditional and it captures most of the benefit.
2. Route on query type. Summarisation, comparison and “read this whole document” requests go long-context. Factual lookups go through retrieval. A small classifier or a cheap model call is enough.
3. Route on retrieval confidence. Retrieve first; if the retrieved evidence scores poorly or the model reports it cannot answer from it, escalate to long context. This is Self-Route’s shape, and it costs an extra call on the escalation path.

Start at 1. It is a conditional, it costs nothing to remove, and it tells you whether the more expensive routers are worth building — if almost every query in your logs selects a corpus on one side of the threshold, options 2 and 3 are solving a problem you do not have.

Where this is heading: agentic retrieval

The direction of travel is away from a single retrieve-then-generate pass and toward iterative retrieval: the model retrieves, evaluates what came back, reformulates, and retrieves again. Query rewriting, hybrid BM25-plus-dense search, and reranking are the components most production systems are assembling. This is a design direction anchored in Self-Route’s routing logic rather than a measured industry statistic — no adoption figures are cited here because none were verified.

What makes it relevant to this comparison is the cost shape. Iterative retrieval trades more calls for smaller contexts per call, which is the opposite trade from long context. It also multiplies query volume against your corpus, which makes the per-query economics above more consequential rather than less, not something to decide after you have chosen an architecture.

What none of this fixes

Choosing between retrieval and long context does not fix retrieval quality, and that is where most production RAG systems actually fail.

If your chunking splits a table across two chunks, the model sees half a table. If your embedding model was trained on general web text and your corpus is medical coding, semantically similar documents will not be nearby in vector space. If you have no reranker, top-k by cosine similarity returns the most similar chunks, not the most useful ones. If you have no evaluation harness, you will not know any of this is happening, because a RAG system with broken retrieval does not throw errors — it returns fluent, confident, wrong answers.

Long context does not fix these either. It removes the retrieval step and therefore the retrieval bugs, and replaces them with position and distraction effects that are harder to debug because there is no retrieved-set to inspect. A wrong answer from a 1M-token prompt gives you nothing to look at; a wrong answer from a RAG pipeline gives you the chunks that were returned, which is a starting point.

This is the same boundary that shows up one layer down in the stack, where giving agents native protocol access to a vector store removes integration code but leaves retrieval engineering the protocol layer does not remove exactly where it was. Architecture choice, access protocol and storage layout are three separate decisions, and none of the three substitutes for getting the right evidence in front of the model.

The work that determines whether your system answers correctly: chunking strategy that respects document structure, an embedding model suited to your domain, a reranker, and an evaluation set you actually run. That work is the same on both sides of this comparison.

Frequently asked questions

Where to go from here

The decision is not RAG or long context. It is which queries in your system belong on which path, and that is answered by three numbers you already have: your corpus size in tokens, your tokens retrieved per query, and your monthly query volume. Run the arithmetic in the worked example with your own figures before the next architecture meeting. If the annual difference is smaller than the cost of building the pipeline, you have your answer and it is not the one this page’s title implies.

The number to be most careful with is the one someone quotes at you confidently without stating its corpus size, its model, its price date or its cache hit rate. That includes the ratios on this page — reproduce them.