Training run killed after gradient spikes to 1.64M and loss hits 12.17
francoisfleuret · x · 2026-11-04
François Fleuret shares a training log that "reads like a description of the Big Bang": a run was stopped and trashed after the first large gradient jump at step 996, gradients reached 1.64 million at step 1004, and loss peaked at 12.17 five steps later.
A vivid case of pretraining instability — from gradient spike to total collapse in about ten steps, underscoring why large-scale runs need spike monitoring and fast kill switches.
More from Fun
- 一条 prompt 在 Lovable 里生成的界面细节惊到网友 — damienghader · 2026-11-05
- Watching lots of AI video makes 'real vs AI' a perceptual property you can flip — dioscuri · 2026-11-05
- Hesamation: frontier labs are in a Mexican standoff, each holding its next model waiting for the others — Hesamation · 2026-11-05
- AI image casts a dad on his recliner as the Emperor of Mankind on the Golden Throne — rickasaurus · 2026-11-05
- ChatGPT found a dead squirrel, not a $1,300 repair: AI saved one homeowner $3,000 in 3 weeks — neil_chilson · 2026-11-05
- Grok Bot has 'totally taken over the show' as Cursor chatter fades — mark_k · 2026-11-05