~65% of TAU-banking agent benchmark tasks found defective
HamelHusain · x · 2026-10-28
danielrupawalla reports finding defects in 65% of TAU-banking tasks, where policies aren't enforced correctly: models can take out-of-scope actions unpunished (tool-call checks only, no DB verification), lie to customers with no penalty (no communication checks), and naive tool-call checks impose no consequences on failures. Shared by Hamel Husain. Recommendation: harden grading before using such benchmarks.
More from Models
- Unreleased model reportedly solves ETH and Quasi-Riemann problems that current models can't — BlancheMinerva · 2026-10-28
- Uncensored LFM3-1.7B GGUF quants drop refusal rate from 61.7% to 0% — alexcovo_eth · 2026-10-28
- CRUX shadow eval: GPT-6 Astra and Claude Fable 5.1 research papers both rejected by experts — patpat_mit · 2026-10-28
- Ollama Launches Cloud Service with Crusoe to Run Frontier Open Models — ollama · 2026-10-28
- How a European AI lab post-training Chinese open models can earn credibility — alejandroll10 · 2026-10-28
- GLM 5.3 Flash NVFP4 on 4x RTX 6000 WS: V2 serving nearly doubles throughput per GPU — TheZachMueller · 2026-10-28