Alibaba's HICO-Bench shows 36% of highly-rated agent conversations fail verification
Tongyi-ConvAI · hf · 2026-10-15
Researchers from Alibaba Token Hub and the University of Tokyo introduce HICO-Bench: 180 everyday coordination scenarios where two agents with private information converse, then independently record the agreed plan. A program checks Feasibility and an LLM judge checks Integrity; passing both is a Verified Success.
Across 13 LLMs, 36% of conversations rated highly for goal completion fail Verified Success. To learn from end-of-conversation outcomes, they propose Paired-Credit Policy Optimization (PaCPO): after each turn the agent proposes a tentative plan, checked against both paired scenario versions, rewarding turns after which tentative plans fit. PaCPO lifts Qwen3-8B's Verified Success from 20.4% to 28.3%.
More from coding & agent
- ASU's τ-Emotion-Bench shows LLM agents stay unreliable when users get emotional — Arizona-State-University · 2026-10-15
- Huawei's SkillMarker watermarks agent skills, detecting copied passages of 1,536 tokens — Alan Tang · 2026-10-15
- GitHub Copilot app adds per-repo sandboxing with folder and network controls — lee_stott · 2026-10-15
- GitHub HydraFusion adds Max tier, model pool now includes GPT-6 family and Claude Opus 5.5 — lee_stott · 2026-10-15
- Xiaohongshu proposes CASI: Shapley-based credit assignment for agentic retrieval — _reachsumit · 2026-10-15
- Fix iOS Safari's ugly browser chrome with two fixed CSS strips — msg · 2026-10-15