EpochAI pegs GPT-6 training at up to 1.8e27 FLOP; scaling01 calls param estimates way too high
scaling01 · x · 2026-10-16
EpochAI, in a New York Times visualization of the math behind AI training, estimates GPT-6 (Astra) pretraining at a median 223M Blackwell GPU-hours (3.75e26 FLOP), RL at 0.19x pretraining FLOP, and total training compute median 9.4e26 FLOP (p95: 1.8e27), with pretraining running a median 88 days on 100K GPUs.
scaling01 argues the total compute estimate is fine but flaws abound: the 5.35x pretraining-to-RL ratio seems silly; since GPT-5.5 (April 23) was trained on Stargate Abilene—likely along with GPT-5.3-Codex, 5.4 and the 5.6 family—Astra's pretraining can't have used all 100K GPUs on average, and probably just ran later (early May–late July); and the parameter estimates ignore sparsity and looping trends, making them obviously too high.
Related event: EpochAI's GPT-6 Compute Scenario Analysis Sparks Debate(3 posts)→
More from Infra
- FreeSolo: Linear-Time Sparse Attention With Zero-Shot Length Extrapolation — Zekun Shi · 2026-10-16
- sleepy-llm: A Reverse Proxy That Auto-Wakes and Shuts Down Your Local LLM Server — ac101m · 2026-10-16
- Where AI chip architecture meets economics: a cross-perspective essay — ai · 2026-10-16
- 797GB of n-grams in read-only CPU RAM match 57% more compute at 1B-scale LM pretraining — anshulkundaje · 2026-10-16
- Retrofitting pretrained transformers with Titan modules speeds up inference — behrouz_ali · 2026-10-16
- NVIDIA Nemotron 3.5 Bolt 74B-A9B Lands on Ollama, Fits 24GB via Native 2-bit Quantization — ollama · 2026-10-16