Mollick: benchmarks tell us little; Agent Arena may be the best we have
emollick · x · 2026-11-16
Ethan Mollick, replying to Nathan Lambert, agrees that current benchmarking is very limited in what it can tell us. Lambert argued closed models are generally more robust, but judging solely on unmeasurable qualities isn't fair — and suggested Agent Arena may be the best available tool for evaluating agents.
More from Models
- ChatGPT external links jump from 10% to 49.5% of conversations in weeks — lilyraynyc · 2026-11-16
- Pangram's 750-word false-positive bounty went unclaimed for 5 days — nobody wants to gamble $1,000 — ctjlewis · 2026-11-16
- Fable 5.2 Tops MazeBench With 21% After 90 Hours and $6,000 in Credits — patience_cave · 2026-11-16
- Tencent Hy4 flash and InternLM Sage-S2.1-Flash both released, tool-calling tests requested — MajesticPuzzler · 2026-11-16
- User Shares Coral Reef Creation Labeled 'Qwen 4 Max (xhigh)', an Unannounced Model Name — intellectronica · 2026-11-16
- Stealth Model 'silicon-2' Surfaces in Devin, Suspected to Be Anthropic's Next Model — kimmonismus · 2026-11-16