matklad's rule of thumb: run micro benchmarks for about 300ms, and here's why
jedisct1 · x · 2026-10-09
matklad (TigerBeetle) lays out a rule of thumb for micro benchmarks: tune input size until a run takes 300ms. Millisecond integers give enough precision to spot small wins; runs under 10ms get skewed by fixed costs like interpreter startup; hundreds of milliseconds sit in the human-perceptible range so intuition works; and it stays fast enough to iterate and eyeball variance across 10 runs. The underlying assumption: benchmarking is about building intuition for decisions, not precise measurement.
More from coding & agent
- Locally: a Grok bot that plans and reviews code while your local model writes it — alexcovo_eth · 2026-10-09
- Crisp-diagrams skill makes coding agents draw brand-styled diagrams from 200+ examples — yenkel · 2026-10-09
- Indie dev trains Rei-v1, a tiny open reranker model for research agents — gnukeith · 2026-10-09
- Small-to-big retrieval is underrated: embed the precise bit, expand after a hit — yi_ding · 2026-10-09
- AgentPaySec seeks feedback on security testing for financial AI agents — Miserable_Gas_1527 · 2026-10-09
- Running Claude and GPT as two AI 'employees' in a tiny company — THINK_NODE · 2026-10-09