Engineer Reasoning Like We Engineer Autopilots: A Control-Systems Manifesto
WillowEmberly · reddit · 2026-08-29
A thoughtful long post asks: why don't we engineer reasoning with the same discipline we apply to systems whose failure can kill us?
The author recalls how aviation engineers built autopilots without modern AI — not by saying "keep the plane where it should be," but by decomposing the problem: attitude, heading, rates of change, deviations, sensor degradation, conflicting signals, correction authority per subsystem, when automation disengages, and how the operator knows the system's believed state. Crucially: what capabilities must survive when components become unreliable?
Reasoning systems share the same structure: estimated states, observations, uncertainty, assumptions, contradictory signals with varying reliability, corrections that overshoot, errors that accumulate, feedback mistaken for confirmation, and errors that erode future error-detection.
Yet we usually just ask whether a person or an AI is "good at reasoning" — the wrong level of abstraction. Better questions: which functions does reliable reasoning require, which must stay independent, how does each degrade, what compensates for its loss, and what preserves recovery when the system's estimate of reality is wrong.
More from AGI Musings
- Opinion: Grok's Rapid Catch-Up Attributed to Distillation Over Compute — teortaxesTex · 2026-08-29
- Living on an exponential: why "this time is different" is always true — phillip_isola · 2026-08-29
- Safety researcher warns labs may soon push a "cyber is solved" narrative — repligate · 2026-08-29
- Is RL on internet-connected, evolving models considered "continuous learning"? — doodlestein · 2026-08-29
- Viewpoint: Biology is nanomachinery, tech has hard limits — sebkrier · 2026-08-29
- Analyzing all 1,329 OpenAI and Anthropic job listings: Personal AGI and robotics are next — imjustnewatai · 2026-08-29