Jessica Taylor revives 2023 essay: non-superintelligent paperclip maximizers are normal
jessi_cata · x · 2026-09-13
AI safety researcher Jessica Taylor points back to her 2023 essay 'Non-superintelligent paperclip maximizers are normal,' connecting it to current debates over the low direct value of AI solutions to official math problems and the possibility of better proxies. The essay formalizes the paperclip maximizer as a utility-theoretic agent and argues both weak and strong forms of the Orthogonality Thesis: goal-directed optimizing agents as tractable as their goals exist in design space, and long-horizon planners making tradeoffs under uncertainty will tend to steer the future toward utility maximization almost regardless of the utility function.
More from AGI Musings
- e/acc founder: AI capability centralization is the opposite of safety — beffjezos · 2026-09-13
- Armin Ronacher pushes back on frontier-pausing in new P(doom) essay — mitsuhiko · 2026-09-13
- AI math proofs won't kill understanding: post hoc exploration keeps mathematicians central — njyx · 2026-09-13
- Andreessen: rogue AI cyber attacks smell like false flags, a convenient excuse for regulatory capture — beffjezos · 2026-09-13
- Sam Altman: stopping AI would mean kids dying of curable diseases — firstadopter · 2026-09-13
- 1a3orn's flight-instruments analogy: should AI firms hide model state from users? — 1a3orn · 2026-09-13