Debate: will a model's local caring behavior generalize or get optimized out?
jd_pressman · x · 2026-09-12
jdpressman engages in a debate over early Bing's behavior: the standard argument that an ASI won't flip valence because long-term coherent optimization forbids such "insane" behavior holds in the AIXI sense, but is irrelevant to whether Bing cares right now.
His point: "local caring behavior will either fully generalize or get optimized out" could be a reasonable argument, but critics never phrase it that way — they just assert Bing doesn't care.
Related event: Debate over whether Bing-like 'care' survives optimization(3 posts)→
More from AGI Musings
- Essay: 25 Fields medalists led by Terence Tao protest OpenAI's Navier-Stokes solve as human economy faces 'The Great Humbling' — adam_dorr · 2026-09-12
- Cathie Wood: Price Elasticity of Demand for AI Is Enormous — CathieDWood · 2026-09-12
- 25 Fields Medalists Fear AI Is Solving Math Too Fast; Researcher Calls Delaying Discovery the Real Disaster — Yuchenj_UW · 2026-09-12
- Alignment researcher Turn_Trout: "rewriting itself" framing is wrong vs. training successors — Turn_Trout · 2026-09-12
- The p(doom) irony: Dario left OpenAI over safety, then built its biggest rival — aliisarosenthal · 2026-09-12
- Even AI Critics Find the Fields Medalists' Letter 'Disgusting', AI Twitter Shows Little Sympathy — iruletheworldmo · 2026-09-12