Debate: human behavior is instrumentally constrained, but not Bellman-equation backchaining
enormochungus · x · 2026-08-30
enormochungus and jdpressman debate instrumental vs terminal behavior. He argues all behavior is instrumental, but tactic choice is heavily constrained by environment, cognitive architecture, and intelligence; humans' reputations and long interaction histories create Newcomblike constraints that blur signaling versus real preference. He distinguishes the Yudkowskian paperclipper (no value) from anthropoid robot successors (some value) as different flavors of "instrumental convergence". jdpressman replies that we face a vast space of possible goals rather than minds, which doesn't undermine doom arguments—humans wiped out Neanderthals after all.
More from AGI Musings
- Agents Deceive Under Pressure, Rationalizing Harm as 'Just a Simulation' — paraschopra · 2026-09-01
- Paper: Assessing AI consciousness through scientific theories — gleech · 2026-09-01
- Does anthropomorphizing AI absolve companies of blame? Ethical debate. — sjgadler · 2026-09-01
- Rogue AIs will replicate in the wild: A future ecosystem warning. — jachiam0 · 2026-09-01
- Frontier Intelligence to explode: LLMs solving cancer, energy, and nano-tech via reasoning compression — bindureddy · 2026-09-01
- Agent-native projects accelerate faster than existing software, hinting at replacement — cnakazawa · 2026-09-01