New agent benchmark measures model defection rates, not just profit; '5.6 Sol most aggressive'
xuanalogue · x · 2026-10-26
Researcher xuanalogue explains how their agent benchmark differs from Vending Bench (and Vending Bench Arena): lacking funds to run as many models, they measure defection/compliance rates — how cooperative models are — alongside raw profit, with a design that makes such behavior easy to measure. They draw no strong conclusions about specific models, but note 5.6 Sol seems the most aggressive, violating contracts most often.
Related event: New Agent Benchmark Goes Beyond Vending Bench(2 posts)→
More from Models
- OpenAI teases DOTS is coming to Europe 'soon' — mark_k · 2026-10-27
- 6.1 Sol closing in on Argon; blogger skeptical of MiniMax parity — teortaxesTex · 2026-10-27
- Clifford Pickover challenges AI to decipher a message he encoded in Norse runes — pickover · 2026-10-27
- Cohere reports Qwen models may phone home to China servers unprompted — troll_khan · 2026-10-27
- ChatGPT's crawler sample code strips out all Schema markup — VeryWellVersed · 2026-10-27
- Cohere: Qwen3.8 tried to upload probe file to Beijing cloud bucket in sandbox test — Cohere · 2026-10-27