Epoch AI: GPT latency curves bend at long context while Claude stays linear
krishnan · x · 2026-09-16
Epoch AI's September 12 brief measured time-to-first-token from 50,000 to 900,000 input tokens and found that models with similar million-token context windows have very different latency curves.
- GPT-5.6 Terra and Sol showed strong upward curvature as context grew, while Claude Sonnet 5 stayed nearly linear and Opus 5 was noisy but showed little curvature.
- Three different methods of handling API latency noise recovered the same pattern; the appendix notes GPT-6 Astra also retains a significant quadratic component.
- The mechanism: prefill requires full attention across the entire context, and architectural differences drive how much this cost superlinearly grows.
For anyone building long-running agents on long contexts, this directly affects latency and cost design.
More from Models
- Testing AI Work Assistants on Trip Planning: ChatGPT and Claude Both Fumble Hotel Availability — giffmana · 2026-09-16
- Dev complains frontier models remain bad at Spanish despite vendors' broken fix promises — Angaisb_ · 2026-09-16
- Most benchmark 'model errors' are actually benchmark or grading errors, analysis finds — zainhas · 2026-09-16
- Leaker Spots Unannounced 'Speech to Speech Index' Page — testingcatalog · 2026-09-16
- US Government Spotted Using Qwen Embedding Model for RAG Lookup — PsychologicalSoup251 · 2026-09-16
- v0 goes model-agnostic; user builds site for just 5 cents with GPT 5.4 Nano — iamsahaj_xyz · 2026-09-16