Model performance degrades in long context; token efficiency varies widely across labs
zakelfassi · x · 2026-08-30
The author argues that large context windows (e.g., 1M) alone are no longer impressive, citing significant performance drops as context grows (e.g., GPT-5.6 Sol dropped from 92.4% at 128K to 61.9% at 512K on needle-in-a-haystack). Different model families show wildly different token efficiencies, even per task. This suggests we are still just scratching the surface of understanding what happens under the hood, potentially leading to 2-3 new emerging disciplines.
More from Models
- 222,753 searches analyzed: how Gemini searches and picks its citations — gaganghotra_ · 2026-10-15
- ModeBench shows RLVR collapses LLMs into single-solution mode; Re:Max training preserves diversity — manoelribeiro · 2026-10-15
- Magnitude open-source inference engine beats Ollama by over 2x with auto-tuned kernels — nickbaumann_ · 2026-10-15
- Hands-on with Google's rumored Gemini 4 Argon: real-time streamed output — vista8 · 2026-10-15
- Game dev head-to-head: Gemini 4 Argon vs. Claude Opus 5.5 — vitaliychiley · 2026-10-15
- Anthropic's Mythos 5.1 pledges to treat Claude's expressed disagreement as decision input — Hesamation · 2026-10-15