Why don't more LLMs ship 10M-token context? A researcher asks about attention tradeoffs
agarwl_ · x · 2026-10-27
Reacting to Fern's claim that its architecture, training, and serving support up to 10M tokens of context—with next-token prediction keeps improving as context grows—researcher agarwl asks a naive but pointed question: why don't we see more major LLM releases with 10M context? Is it because they'd have to switch away from quadratic attention to non-quadratic variants?
More from Models
- 4 Hours of LLM Startup Research Produced a Convincing but Nonsense Product Thesis — annetgriffin · 2026-10-27
- Dev argues Claude burns far more tokens than GPT models for the same coding work — haider1 · 2026-10-27
- MiniMax matches Claude Opus 4.5 in non-benchmaxxed evals, users report — teortaxesTex · 2026-10-27
- Real-world math: GPT-6.1 Terra's half price is offset by 2-3x token usage, costing more than Sol — lordpuddingcup · 2026-10-27
- Fable 5.5 rumored to ship this week as partners still test — kimmonismus · 2026-10-27
- GPT 6.1 Terra vs Claude Sonnet 5.5: speed and value vs raw capability — teortaxesTex · 2026-10-27