Lost in the Middle: How Language Models Use Long Contexts
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang
cs.CL
2023-07-07
Moving the gold passage through a long prompt yields a U-curve; GPT-3.5-Turbo falls below its 56.1% closed-book score in the middle, and a longer window barely helps.
By 2023, models advertised 4K, 16K, even 100K-token windows. Retrieval-augmented generation and long-document QA started dumping full search-result lists into the prompt, and a longer window was often treated as a stronger model. This paper asks a narrower question: if you move the relevant span inside that window, does accuracy hold.
If long context were used robustly, putting the answer in document 1 versus document 15 should barely matter. It matters a lot, and the middle is the worst place.
Two controlled tasks.
Multi-document QA uses 2655 NaturalQuestions-Open items whose annotated long answer is a paragraph. The prompt contains exactly one gold passage and k−1 Contriever distractors that do not contain the annotated answers. k is 10, 20, or 30, about 2K / 4K / 6K tokens. Reordering moves the gold passage; extra distractors lengthen the prompt. Accuracy is whether any annotated answer string appears in the generation.
Key-value retrieval is cleaner: a JSON object of random UUIDs, and the model must return the value for a given key. k is 75, 140, or 300 pairs, about 4K / 8K / 16K tokens. Almost no natural-language semantics, so it tests exact match retrieval from context.
Open models are MPT-30B-Instruct (8K) and LongChat-13B (16K). Closed models are GPT-3.5-Turbo (4K and 16K) and Claude-1.3 (8K and 100K). Decoding is greedy. Flan-T5-XXL and Flan-UL2 probe encoder-decoder models. Base MPT-30B probes whether instruction tuning created the pattern.
Multi-document QA traces a U-shaped curve: best at the start or end, worst in the middle.
| Model | Closed-book | Oracle | 20 docs, gold first | 20 docs, gold middle |
| GPT-3.5-Turbo | 56.1% | 88.3% | 75.8% | 53.8% |
| GPT-3.5-Turbo (16K) | 56.0% | 88.6% | 75.7% | 54.1% |
| Claude-1.3 | 48.3% | 76.1% | 59.9% | 56.8% |
| LongChat-13B (16K) | 35.0% | 83.4% | 68.6% | 55.3% |
With 20 documents, GPT-3.5 drops from 75.8% at the front to 53.8% in the middle, below its 56.1% closed-book score. At 30 documents the 16K variant hits 50.5% in the middle. The 4K and 16K curves nearly overlap, so a longer window does not by itself teach the model to use context.
Claude-1.3 is near-perfect on key-value retrieval. GPT-3.5 and MPT sag in the middle at 140 and 300 pairs. Putting the query both before and after the data lifts key-value retrieval to near-perfect and barely moves multi-document QA.
Flan-UL2, inside its 2048-token training length, varies only 1.9 points from best to worst; beyond that length the U returns. Base MPT-30B already shows the U. Instruction tuning shrinks the gap from about 10 points to about 4, and the shape remains. Llama-2 7B is mostly recency-biased; 13B and 70B show both primacy and recency.
In open-domain QA, retriever recall keeps rising with k while reader accuracy flattens after about 20 documents: roughly +1.5 points for GPT-3.5 and +1 point for Claude going from 20 to 50 docs.
A context-window marketing number is not the same as using those tokens. In RAG, fetching more passages can hurt: more distractors, and the useful one is more likely to sit in the middle. Reranking gold-ish passages to the edges, or truncating the list, is the more practical lever.
Long-context eval should not be a single average. The protocol here is strict: best-case and worst-case need to be close before anyone claims robust use of long context.
The tasks are clean, and therefore laboratory-like. Multi-document QA always has exactly one gold passage; real retrieval often has zero or several. Distractors are sorted by relevance, which may interact with a "search results" prior, though a random-order appendix still shows the same shape. Exact-match scoring misses paraphrases. GPT-4 is only a subset. All models are mid-2023; later long-context training recipes may flatten the U, and this paper cannot say. Query-aware prompting saves key-value retrieval and not QA, so "cannot find" is only part of the story. "Finds it and still fails" is the rest, and the mechanism is open.