Alignment debate: LLM pretraining prior may weight human-compatible goals heavily
EigenGender · x · 2026-10-29
EigenGender pushes back on the claim that almost all goals in the relevant prior are incompatible with human survival, calling it a type error: the LLM pretraining prior plausibly places substantial weight on human-goal-matching entities. He also frames the deeper cruxes — whether approaching superintelligence necessarily means coherent goal-seeking agents, and whether standard ML methods can instill goals safely.
Related event: EigenGender dissects the core cruxes of AI alignment debates(5 posts)→
More from AGI Musings
- Ex-Insider Questions Anthropic's Claim That "AI Safety's Technical Problems Are Solved" — ZhongRuiqi · 2026-10-29
- Pedro Domingos: OpenAI and Anthropic are the WorldCom and Global Crossing of AI — pmddomingos · 2026-10-29
- Opus videos that wowed two weeks ago already look like slop — our hunger is infinite — jzlegion · 2026-10-29
- Burny: AI progress is more continuous than most assume — it's continuous hillclimbing — burny_tech · 2026-10-29
- Pedro Domingos: AI's enormous value will almost all be captured by users — pmddomingos · 2026-10-29
- Dune's no-computers rule isn't a plot hole — it's the Butlerian Jihad — SumitGup · 2026-10-29