Measuring agents by wall-clock time: harness and multi-agent overhead matter
rickasaurus · x · 2026-10-17
In a discussion on agent evaluation, rickasaurus argues that harness overhead and multi-agent orchestration overhead must also be measured, not just task success rates.
His proposed metric: wall-clock time per model to build a useful thing at some very strict standard — judging agents by real delivery speed rather than benchmark scores alone.
More from coding & agent
- shadcn reflects: it's only been a year since Claude Skills launched — shadcn · 2026-10-17
- ClinOpsBench: 15 AI agents tested on 4-month simulated clinical trial oversight — ycombinator · 2026-10-17
- PR Lens matches agent PR screenshots to architecture diagrams in ~2s, 2k GitHub stars — nevrekaraishwa2 · 2026-10-17
- Cohere research: agent teams hit frontier performance at 30% of the cost — nouhadziri · 2026-10-17
- Four agent frameworks shipped the same fix in one week: context compaction vs approvals — HelpfulCollar1070 · 2026-10-17
- Pipecat's open PhoneLLM Max Alpha 2 tops voice-agent bench at 83%, beats Sonnet 5.5 — solyarisoftware · 2026-10-17