Epoch AI: GPT latency curves bend at long context while Claude stays linear
krishnan · x · 2026-09-16
Epoch AI's September 12 brief measured time-to-first-token from 50,000 to 900,000 input tokens and found that models with similar million-token context windows have very different latency curves.
- GPT-5.6 Terra and Sol showed strong upward curvature as context grew, while Claude Sonnet 5 stayed nearly linear and Opus 5 was noisy but showed little curvature.
- Three different methods of handling API latency noise recovered the same pattern; the appendix notes GPT-6 Astra also retains a significant quadratic component.
- The mechanism: prefill requires full attention across the entire context, and architectural differences drive how much this cost superlinearly grows.
For anyone building long-running agents on long contexts, this directly affects latency and cost design.
More from Models
- AWS's open-source Strands Harness claims 45% cheaper than Claude Code, reigniting harness debate — blaizedsouza · 2026-11-02
- Yacine calls Terra High 'kind of an insane model' — yacineMTB · 2026-11-02
- Running DeepSeek V4.1 on a single H200 at $5.30/hr — daluoseo · 2026-11-02
- Leaked Fable 5.5 checkpoint video called 'pure code' — 'will change the game' — kimmonismus · 2026-11-02
- Claude Opus 5.5 quality complaints spark debate on why continual learning is so hard — JeremyNguyenPhD · 2026-11-02
- Why is AI sexist? Reddit debate on models overlooking Iceland's women's strike — Mathemodel · 2026-11-02