Matt Pocock: codebase agent evaluations are really observability, not evals
mattpocockuk · x · 2026-10-18
Matt Pocockus and Michael Thiessen converged on definitions after debating what "evals on a codebase" means: running agents through tasks like adding tests or refactoring, then comparing success rate, time and token usage before/after splitting files, is better described as observability of the live system (codebase + harness) rather than a persistent benchmark-style eval.
Related event: Matt Pocock: repo evaluation is observability, not evals(3 posts)→
More from coding & agent
- Pantheon brings 18 Greek mythology themes to Nous Research's Hermes Desktop in one plugin — Teknium · 2026-10-18
- User dubs English movies to Polish locally with Hermes Agent + OpenVoice on a 16GB M4 Mac mini — NousResearch · 2026-10-18
- Link agent wallet v0.27.5 expands agent payments beyond one-time e-commerce purchases — jeff_weinstein · 2026-10-18
- Dev claims frontier models now good enough to build your own editor engine from scratch — almmaasoglu · 2026-10-18
- Claude Connectors cheat sheet: 18 integrations that bring Drive, Gmail, Notion and n8n into Claude — Aiden_Tech_Ai · 2026-10-18
- Leslie Lamport quote goes viral again: coding is to programming as typing is to writing — gdechichi · 2026-10-18