When should AI agents change their mind from pre-specified preferences?
jacyanthis · x · 2026-11-15
In a follow-up to his agent preference-representation experiments, jacyanthis raises the deeper question: I'd want my agent to 'change its mind' from my pre-specified preferences when it sees new evidence — but how should it decide? Should agents simulate their human and only deviate when they expect the human would post hoc support the change? This gets at a core design problem in agent representation: static preference snapshots versus dynamic, evidence-updated preference modeling.
More from Safety
- Gary Marcus pushes back: I never denied AI risks — my 2023 TED talk has 1.6M+ views — GaryMarcus · 2026-11-16
- Safety researcher: current models already pose grid-down cyber risks to infrastructure — gmiller · 2026-11-16
- Human oversight is broken: monitoring systems are themselves misaligned in correlated ways — davidmanheim · 2026-11-16
- Cisco Talos warns AI-powered pentesting frameworks like ARTEX are automating full attack chains in the wild — cyb3rops · 2026-11-16
- Microsoft: Threat actors use agentic ARTEX framework to steal cloud secrets in 5 minutes — cyb3rops · 2026-11-16
- Stop installing skills and MCP servers from viral threads unread — weswinder · 2026-11-16