Michael Thiessen shares a practical method to benchmark coding agents on your codebase
MichaelThiessen · x · 2026-10-18
Responding to "what do evals on a codebase even mean", Michael Thiessen outlines a concrete method: have your agent run several tasks (adding tests, refactoring, etc.), compare the codebase before and after splitting files into smaller modules, then track success rate, time and token usage — and inspect traces to see whether agents navigate the code more efficiently or get more confused.
Related event: Matt Pocock: repo evaluation is observability, not evals(3 posts)→
More from coding & agent
- Musk amplifies Grok Bot that phones restaurants and books tables for you — elonmusk · 2026-10-19
- Build a calibrated decision model on any LLM without retraining, Colab inside — kalyan_kpl · 2026-10-19
- Melony adds a free chat UI to share Claude Managed Agents with your team — DarasStayHome · 2026-10-19
- How 90% of software engineering looks in 2026: engineers just pass PRs between the same AI — Yuchenj_UW · 2026-10-19
- PM drowning in 10 routing pings a day builds a 'Company OS' in Slack that answers with sources — aakashgupta · 2026-10-19
- Codex now suggests follow-up prompts — the machine tells you what to ask next — burkov · 2026-10-19