ASCI Q supercomputer crashed every 6.5 hours until a neutron beam exposed cosmic rays as the culprit
lauriewired · x · 2026-09-29
lauriewired revisits the reliability crisis of ASCI Q, the world's #2 supercomputer in 2002-2003: it failed every 6.5 hours on average, full-machine reboots took up to 8 hours, and utilization sat at only 60%, with rare "few good weeks" above 85%.
Engineers suspected cosmic rays causing uncorrectable bit flips. At Los Alamos they pointed their own neutron beam at the machine—5 seconds of beam time equaled 6+ years of normal sun exposure—and traced the failures to missing correctable parity in the L2 cache of the Alpha CPUs. Later machines adopted ECC-protected L2 plus better software fallbacks and checkpointing.
Takeaway: when pushing the frontier, you sometimes have to work with very unreliable systems.
Related event: ASCI Q Supercomputer Crashed Every 6.5 Hours(2 posts)→
More from Fun
- Philip K. Dick's 1955 Autofac: humans fight an automated economy with prompt injection — moultano · 2026-09-29
- SF Rationalists professed AI safety and p(doom) — until they confessed they only love sex parties — jjvincent · 2026-09-29
- AI-generated action figure fight: Mega Muscle Martha VS MogMog — KloneStudios · 2026-09-29
- AI Circle Joke: Alexander Wang Went From Fashion Empire to AI — djcows · 2026-09-29
- Meta's 'Sam 3.1' Version Number Sparks Delighted Confusion on X — threepointone · 2026-09-29
- Developer Joke: Slowly Becoming the AI Agent's IRL Tool Call — dreamwieber · 2026-09-29