Too many 'X times faster' claims hide extremely narrow benchmarks, says Ruff author
charliermarsh · x · 2026-09-25
Ruff creator Charlie Marsh adds to the microbenchmark critique: too many projects claim "I made X Y times faster," but clicking through reveals the speedup applies only to an extremely narrow use case, far from the general performance image their sites project.
More from coding & agent
- Cursor mulls killing Plan Mode for a Shift+Tab effort-level hotkey — steipete · 2026-09-25
- Dev burns 5-10B tokens a day running 24 Devin agents plus Codex and Claude Code — teropa · 2026-09-25
- Question's Gambit tops BrowseComp-Plus recall with 96.6% using only BM25 — CShorten30 · 2026-09-25
- Engineer tests now bait AI tools with plausible-but-wrong answers to grade verification skills — l4rz · 2026-09-25
- AutoScientists NeurIPS paper: self-organizing AI research teams hit 74.4 percentile on BioML-Bench — marinkazitnik · 2026-09-25
- The PM role is now 3 jobs: AI dissolves barriers between customer, engineering, and business — aakashgupta · 2026-09-25