ASU's τ-Emotion-Bench shows LLM agents stay unreliable when users get emotional
Arizona-State-University · hf · 2026-10-15
- Motivation: in long-horizon tool-calling, agents fail not only on tool use but also on ineffective user interaction; existing benchmarks assume cooperative, stable users.
- Benchmark: τ-Emotion-Bench spans ten domains and evaluates tool-calling agents under controlled emotion-conditioned user interactions while preserving task objectives.
- Tracer pipeline: a scalable agentic data-generation pipeline that builds grounded, policy-valid tasks and collects trajectories via an interactive agent-user-environment harness; dissatisfied users can terminate inefficient interactions, filtering trajectories with excessive user dependence.
- Findings: frontier agents remain unreliable, with large gaps between average task success and consistent completion; Tracer-generated supervision consistently improves Qwen3 models across scales, generalizes to unseen domains and transfers to τ-Bench and τ-Trait; higher success correlates with better balance between tool execution and necessary user follow-up.
More from coding & agent
- SWEeper-Bench: Best of 15 Frontier Agents Solves Only 59% of Interactive Bug-Finding Tasks — zlab-princeton · 2026-10-15
- User hands Google login and SMS 2FA to AI agent, reigniting credential-handoff debate — jonstephens85 · 2026-10-15
- Grok Bot writes its own Apple Desk skill to control Mac apps, Musk amplifies — elonmusk · 2026-10-15
- Thorsten Ball: latest models make context-management tricks like /btw and forking obsolete — andreisavu · 2026-10-15
- User says Grok Bot finished 10 days of work in under 2 hours on a $30 plan — elonmusk · 2026-10-15
- PersonalAgentEval launches: 7 personal agents tested on real 13-month histories — HennyJieCC · 2026-10-15