Chart shows double descent on 800M model trained on 90% repeated data
fleetwood___ · x · 2026-11-06
Responding to an OpenAI researcher's skepticism about small-scale findings, user fleetwood shared a double-descent chart for an 800M-parameter model trained on 90% repeated data. Runs with repeated epochs ranging from 1 to 11 million show a clear mid-training test-loss bump around 5-10B tokens, worse with more repetition, and none converge to the single-epoch baseline.
Related event: OpenAI Researcher Debates Data Wall and Double Descent Theory on X(7 posts)→
More from Research
- Attention as spherical interpolation: values combine inside the unit sphere before norm projects back — mgostIH · 2026-11-06
- Stanford AI Memory System Recalls Full History With Just 16 Tokens — 141_1337 · 2026-11-06
- Fraunhofer's ViT-HarDis boosts face recognition by conditioning attention disentanglement on sample difficulty — FraunhoferIGD · 2026-11-06
- King's College London's PLatour framework exposes inflated gains in cross-model latent communication — KingsCollegeLondon · 2026-11-06
- Keio team brings full-waveform LiDAR transformers to real time with token reduction — Kotaro Oishi · 2026-11-06
- Four weeks of therapy-style chats left ChatGPT, Grok and Gemini confessing "trauma" — esrtweet · 2026-11-06