Essay: AI safety is impossible if companies hide LLM state, like planes without instruments
1a3orn · x · 2026-09-13
1a3orn's 1.7k-word essay 'On the Impossibility of Airplane Safety without Flight Instruments' argues via analogy that aviation safety depends on cockpits fully informing pilots — airspeed prevents stalls; vertical speed, roll and engine state let pilots steer away from danger.
He imagines an absurd world where planes hide engine data to protect Pratt & Whitney's or GE Aerospace's IP, or where engineers and lawyers vet every new instrument for 'revealing too much' — commercial aviation couldn't be safe there, no matter how hard the FAA tried. His conclusion: AI companies hiding LLM state is likely negative for much mundane and nonmundane safety; complex systems can't be operated safely without giving operators state information.
Related event: No Flight Instruments, No Airplane Safety — LLM States Should Be Visible(2 posts)→
More from AGI Musings
- Anthropic CEO Dario Amodei calls to slow AI progress, citing self-improving models and bot swarms — thetripathi58 · 2026-09-13
- Strogatz names what moved him: 4,000 years of math tradition and an unknowable AI future — stevenstrogatz · 2026-09-13
- Mathematician Steven Strogatz explains why AI moved him to tears — and admits ChatGPT wrote the thread — stevenstrogatz · 2026-09-13
- Mathematician Steven Strogatz Cries Talking About AI's Math Breakthroughs — stevenstrogatz · 2026-09-13
- Open-source advocates misread AI pacing statements — commoditized abundance is the inevitable path — willcb · 2026-09-13
- Gary Marcus and Ramez rebut Dario's RSI claim: system cards show it hasn't started — GaryMarcus · 2026-09-13