Agent benchmark: two fastest models finished zero of 12 tasks
ML_Felix · reddit · 2026-10-15
Data from Concourse Bench v1 (author works at the publisher), where agents negotiate a simulated media buy against three fixed-rule sellers with per-action validation: Haiku averaged 0.38 min per brief but completed 0/12 tasks; Luna 1.01 min, 0/12; Sol 3.63 min, 12/12; Fable 8.89 min, 12/12; Sonnet 13.26 min, 9/12. The fastest models were fast because they gave up — yet the slowest wasn't best either. The thread asks what actually matters when judging agents: speed, completion rate, or cost per finished task.
More from coding & agent
- Claude cut a 2.6-hour video export to 20 minutes — but only after being told to rebuild the server — byhewar · 2026-10-23
- 3 flight-booking rules my AI agent checks that I'd never do by hand — Efistoffeles · 2026-10-23
- OpenJet claims Qwen3.8-27B coding runs 1.9x faster than llama.cpp on one RTX 3090 — JeremyCMorgan · 2026-10-23
- Building an autonomous SEO agent in Hermes with three skills: programmatic pages, Keywords Everywhere, DataForSEO — EXM7777 · 2026-10-23
- Building a sports betting AI agent with onchain execution and human approval by default: architecture and lessons — Dry-Comfortable-2678 · 2026-10-23
- How do you audit AI agent decisions in regulated workflows? A field survey — Ok_Way_1993 · 2026-10-23