I ran 16 models to vet one tool: one task is not a benchmark
AlexKim · x · 2026-09-19
To decide if TypeSafe's Jev belonged in his stack, the author benchmarked 16 models — Jev came 10th on accuracy. His first draft claimed "Haiku was more accurate," true on one task only; adding two more made it a split: Haiku wins business categories 83.2–79.9, Jev wins commit types 50.0–42.0, dead tie on prose at 66.0. Lesson: one task is not a benchmark.
More from coding & agent
- Veteran game dev: "AI builds a full game in a day" posts are mostly hype — bendee983 · 2026-11-04
- Dev builds interactive Excel-to-dashboard with ChatGPT, shares the prompt — TermRemarkable665 · 2026-11-04
- Agents cut dev time from weeks to a day; bottleneck moved to specs and review — Suspicious_Orchid770 · 2026-11-04
- Orgo, a YC-backed platform giving AI agents their own cloud desktops, claims 1,000+ teams — Scobleizer · 2026-11-04
- Founder: Vibe-coded a PMF-successful product, then hired pro devs to fix the mess — sujingshen · 2026-11-04
- One prompt to audit cross-region traffic: dev cut trans-Atlantic latency by moving a service — DanielLockyer · 2026-11-04