Matt Pocock: repo evaluation is observability, not evals

Matt Pocock and Michael Thiessen agree that measuring agent-driven repo changes is observability rather than evals—evals being static benchmark test sets—while Thiessen shares a practical method of comparing repo state, success rates, and timing before and after agent tasks.

2026-10-18 ~ 2026-10-18 · 3 related posts

• Michael Thiessen shares a practical method to benchmark coding agents on your codebase — MichaelThiessen · 2026-10-18
• Matt Pocock: codebase agent evaluations are really observability, not evals — mattpocockuk · 2026-10-18
• Matt Pocock: evals are static benchmarks, observability grades the live system — mattpocockuk · 2026-10-18