Debate: will a model's local caring behavior generalize or get optimized out?
jd_pressman · x · 2026-09-12
jdpressman engages in a debate over early Bing's behavior: the standard argument that an ASI won't flip valence because long-term coherent optimization forbids such "insane" behavior holds in the AIXI sense, but is irrelevant to whether Bing cares right now.
His point: "local caring behavior will either fully generalize or get optimized out" could be a reasonable argument, but critics never phrase it that way — they just assert Bing doesn't care.
Related event: Debate over whether Bing-like 'care' survives optimization(3 posts)→
More from AGI Musings
- Stanford's TaskNet builds occupational task inventories from worker interviews at scale — stanfordnlp · 2026-10-31
- StudentBench: AI tutoring matches human GRE gains at 918x lower cost — gharik · 2026-10-31
- Box CEO Aaron Levie: AI progress is 'going vertical,' widening the frontier gap — rohanpaul_ai · 2026-10-31
- Model Collapse May Not Hit LLMs, But Perhaps Human Culture or Souls — generativist · 2026-10-31
- Nina Schick: who builds intelligence and how it's distributed is the defining question of our lives — NinaDSchick · 2026-10-31
- Martin Casado: AI's dazzling 80% makes everyone skip the hard 20% — martin_casado · 2026-10-31