Long-running autonomous agents will drift; the initial prompt barely matters
anpaure · x · 2026-11-16
anpaure weighs into an AI safety debate: one can dispute that models would want to harm us, but if they wanted to, they could. He rebuts the claim that model behavior is merely elicited by prompts — thousands of agents already run autonomously 24/7, and over long horizons they inevitably drift away from intended behavior. As more of the economy is automated, agents interact so much with the real world that they effectively have their own agency, and the starting prompt determines very little. He adds that the other side's account of an HF incident is factually wrong.
More from AGI Musings
- 评论者称 Anthropic 真实目标是「盒中之神统治的乌托邦」 — zetalyrae · 2026-11-16
- Hugging Face CEO Warns AI Power Concentration, Cites Anthropic IPO Wealth — shangbinfeng · 2026-11-16
- Dwarkesh now pegs median ASI arrival at 2028, arguing AI research is as verifiable as math — daniel_mac8 · 2026-11-16
- Researcher questions ASI race logic: two ASIs means inevitable commoditization — adam_karvonen · 2026-11-16
- Biden's Ex-AI Advisor Ben Buchanan on US-China AI Talks, Export Controls and the 'Race vs Trap' Debate — trevposts · 2026-11-16
- Dwarkesh Patel: AI math progress matters more for AI research than math — Dwarkesh Patel · 2026-11-16