60% of chatbot responses miss critical safety warnings — what does that mean for autonomous agents?

fortiplus84 · reddit · 2026-10-22

An NYU / UW–Madison study tested four AI chatbots on technology-facilitated domestic abuse questions and found 60% of assessed responses omitted critical safety warnings.
Example: a bot suggests changing passwords or device settings without recognizing this could alert an abusive partner. The advice is technically correct but the model misses real-world consequences.
The poster extends this to autonomous agents: when an agent can execute actions, modify settings, and touch external systems, mistakes become hard-to-reverse actions rather than misleading advice.
Open questions: should agents be required to detect high-risk situations and seek human approval? Can better reasoning, contextual safeguards and tool-level restrictions suffice without making agents unusable?

More from AGI Musings