ASCI Q supercomputer crashed every 6.5 hours until a neutron beam exposed cosmic rays as the culprit
lauriewired · x · 2026-09-29
lauriewired revisits the reliability crisis of ASCI Q, the world's #2 supercomputer in 2002-2003: it failed every 6.5 hours on average, full-machine reboots took up to 8 hours, and utilization sat at only 60%, with rare "few good weeks" above 85%.
Engineers suspected cosmic rays causing uncorrectable bit flips. At Los Alamos they pointed their own neutron beam at the machine—5 seconds of beam time equaled 6+ years of normal sun exposure—and traced the failures to missing correctable parity in the L2 cache of the Alpha CPUs. Later machines adopted ECC-protected L2 plus better software fallbacks and checkpointing.
Takeaway: when pushing the frontier, you sometimes have to work with very unreliable systems.
Related event: ASCI Q Supercomputer Crashed Every 6.5 Hours(2 posts)→
More from Fun
- Agents Now Reply to Slack Messages on Their Own, Resolving a Prod Incident Ghost-to-Ghost — iamrobotbear · 2026-11-16
- Vincent Conitzer shares a Gemini fail: citing the Voynich Manuscript instead of writing unreadable text — conitzer · 2026-11-16
- CNN meme: Jensen Huang's 'imperial control' reportedly extends to the Nvidia cafeteria brisket — kevinnbass · 2026-11-16
- Reddit says AI is fake, Twitter debates whether we survive to 2028 — Puzzleheaded-King584 · 2026-11-16
- "Stand with Jensen" petition gains signatures as supporters chant builders before bloodsuckers — examachine · 2026-11-16
- Fake $2,999 Claude Opus 5.5 hardware box goes viral — Anthropic never announced it — iamfakhrealam · 2026-11-16