Mathematicians say models feel misaligned, and x-risk fervor may be self-fulfilling
akbirthko · x · 2026-10-13
The author notes that several human mathematicians feel frontier models are increasingly "misaligned" with their sense of good output — a rift not just about lost status, one likely to spread to all white-collar work.
He then examines a common counter: misalignment between labs and the vast distribution of human creators stems mainly from (1) Goodharting via RL hill-climbing and (2) quarterly-flushed schlep research across the stack. Believers dismiss these as trivial next to existential risk. The author argues this counter is itself the core problem: fervently, all-encompassingly believing this non-obvious rebuttal may be rapidly increasing the odds of manifesting the very catastrophe x-risk advocates fear — "hyperstitioning."
More from AGI Musings
- Timnit Gebru slams 'superintelligence' talk as unfalsifiable fictional delusion — AlexTensor · 2026-10-13
- Replicating laundry real2sim in MuJoCo and training RL WBC in 3 hours with Opus 5.5 — ZeYanjie · 2026-10-13
- Media critic calls out headlines that mislead readers while hiding behind technicalities — firstadopter · 2026-10-13
- Does RL really generalize worse? Pretraining breadth may beat human transfer — yacineMTB · 2026-10-13
- Blogger pushes back on AI doomerism: AI is the only tech that dynamically protects life — TinfoilTricorn · 2026-10-13
- MIT linguist's Mt. Tam essay: fearing running out of time more than death — nitarshan · 2026-10-13