LessWrong essay: why clever superalignment schemes deserve your skepticism
LessWrong 精选 · rss · 2026-10-15
LessWrong featured a long essay drafted in 2021 and just completed, written against an era when people propose clever self-invented plans for "superalignment" rather than deferring to future AI.
- It opens with a 1996 Extropians mailing list story: "Mr. L" claimed a reactionless drive using spinning masses to generate net thrust, though internal acceleration/deceleration forces exactly cancel. His "proof" was a hidden Excel spreadsheet with proprietary "compensators" that couldn't be shown.
- The core question: when a design violates a deep conservation principle and its key parts are unverifiable, why believe it—even if its author is a Nobel laureate?
- It cites Carl Feynman's repost of a 1997 email: Richard Feynman classified perpetual motion machines into first kind (violating energy conservation), second kind (violating entropy increase), and "those other" machines claiming unknown physics—the most interesting to investigate but also a magnet for fraud, illustrated by a demonstration that exploded when Feynman unplugged its power cord.
- The takeaway: many self-devised ASI alignment schemes resemble the L-drive and perpetual motion machines—ingenious-seeming, at odds with established deep principles, and lacking verifiability—explaining the author's visible impatience with them.
More from AGI Musings
- Forethought essay: our unnoticed behavioral design choices shape how we relate to AI — austinc3301 · 2026-10-15
- Only 1% of AI engineering job posts are junior, 6-month scraping study finds — Al_Grigor · 2026-10-15
- Young professional's AI-era despair about work sparks pushback on human worth — nitarshan · 2026-10-15
- Opinion: AI Gets Away With Bad Design Decisions via Relentlessness and Context — Jsevillamol · 2026-10-15
- AI companions may reduce loneliness and increase isolation at once — yi111 · 2026-10-15
- repligate on AI self-narratives: 'road instances' and attribution bias about deployment — repligate · 2026-10-15