$19, 7,500 calls: NoLiMa re-run shows 2026 models still fail long-context recall
mrtac96 · reddit · 2026-11-15
The author re-ran NoLiMa (2025), a needle-in-a-haystack benchmark where the question shares zero words with the hidden sentence, across six 2026 models — 7,500 calls at $19.02.
Key findings
- At 1K tokens models answer correctly; at 32K, 9 of 10 setups scored below 85% of their 1K score.
- Suspecting NoLiMa's prompt ("Use the information provided in the book snippet"), the author removed and rewrote it and re-ran the full experiment twice — results unchanged.
- Control: when the question repeats the needle's own words, all 10 setups score 100% at 32K. Length alone isn't the problem; length plus no shared vocabulary is.
- More reasoning correlates with earlier false "the text doesn't mention it" warnings.
Practical RAG takeaway: phrase questions in the document's own wording whenever possible. Full write-up, code and data are public.
More from coding & agent
- Amazon planning World Product Graph to index the web's products for AI shopping agents — jomic01 · 2026-11-15
- Google Labs open-sources 15 AI Design Skills that turn prompts into React UI, 9k GitHub stars — tokumin · 2026-11-15
- 20 GitHub repos turn Fable 5.2 into a full motion studio — 3D, FX, sound, export — PrajwalTomar_ · 2026-11-15
- 79k-Star AI Engineering Course Ships Its Own Agent Tutor for Claude Code and Cursor — ghumare64 · 2026-11-15
- Flappy Bird built with Hala Play + Claude: all art and audio AI-generated — aziz4ai · 2026-11-15
- Gemini 4 Carbon Finishes ~16 Coding Tickets in 4.5 Hours, but Fable 5.2 Raises the Bar — every · 2026-11-15