50 agent runs, zero doc reads: five models pass tests but skip the documentation
ducha_aiki · x · 2026-10-26
An agent benchmark across five models (per the attached image): Sonnet 5.5, Flash, Luna, and Terra passed 10/10 while Flash-Lite managed 7/10—but completion times varied wildly, with Sonnet 5.5 at a 24s median vs Flash at 9.7 minutes. The striking finding: across 50 runs, no model ever fetched a documentation page. "Tell me agents are not like people"—they rush to finish without reading the manual.
More from coding & agent
- Kubernetes works for vLLM inference, but agent sandboxes needed Modal's custom stack at 1M+ concurrency — NielsRogge · 2026-10-26
- badlogic on his 2026 agent workflow: "I tell them what to do, I yell when they do it badly" — steipete · 2026-10-26
- Indie dev's trip planner fails 3 AI review rounds before making offline prices read-only — SeanOliver · 2026-10-26
- $21 of DeepSeek API wrote Molten: a Rust+Metal engine serving 300+ tok/s on M5 Macs — GabGarrett · 2026-10-26
- EMNLP 2026 Findings: "Ask or Assume?" turns coding agents into proactive help-seekers — najoungkim · 2026-10-26
- Purchasing agent in production: >95% recall, >90% accuracy, author asks how to push F1 higher — Ancient_Ad_916 · 2026-10-26