OpenAI researcher: 800M-model intuitions are useless for GPT-6-scale training
willdepue · x · 2026-11-06
OpenAI researcher Will DePue pushed back on extrapolating small-model training findings to frontier scale, arguing that intuitions from tiny models are useless for thinking about GPT-6-sized runs — an 800M model trains in 90 seconds on a single H100 node.
His interlocutor countered that the phenomenon (loss behavior under heavy data repetition) has been observed at larger scale, citing a chart of double descent on an 800M model trained on 90% repeated data: runs with repeated epochs from 1 up to 11 million show a pronounced mid-training loss bump and never converge to the no-repetition baseline. DePue added that the papers cited each describe very different phenomena.
More from Models
- OpenAI's AI finds 948 working Linux kernel privilege escalation bugs for $1.2M in API credits — Fcking_Chuck · 2026-11-06
- Custom GPTs die Dec 11: if you use someone else's GPT, you can't migrate it — Ok_Negotiation_2587 · 2026-11-06
- Thinking Machines Ships Inkling 2 Small: 68-Hour Pretrain, Top Non-China Open-Weights Model — Latent Space · 2026-11-06
- Kimi K3.1 and Tencent Hy4 accused of gutting cybersecurity skills from open-weight models — Fresh_Sun_1017 · 2026-11-06
- On the METR graph: insiders seeing AECI tasks is weak evidence, not zero evidence — gleech · 2026-11-06
- Daily AI brief: GPT-6.2 imminent, Claude Personal Finance beta, METR says Anthropic progress up 1.9x — testingcatalog · 2026-11-06