After reading 46 resources on agent evals, one author explains why her 95% score was misleading
blaizedsouza · x · 2026-11-02
After going through 46 resources on agent evals, the author wrote a comprehensive article triggered by a telling experience: her agent scored 95% on evals she wrote herself, yet she knew something was wrong before she could articulate it. The piece systematizes the core knowledge of agent eval design — why self-authored evals self-deceive, how to build eval sets that expose real weaknesses, and common pitfalls.
More from coding & agent
- Xiaomi MiMo's CodeSpecWeave trains coding agents on interdependent specs for long-horizon coding — XiaomiMiMo · 2026-11-02
- Sketchy v0.1.6: AI diagrams use 29% fewer output tokens than plain text — Nkgsporty · 2026-11-02
- Avoiding AI brain rot: Miles Cranmer's lecture on learning with AI — _ScottCondron · 2026-11-02
- Scrapling web scraping framework tops 92k GitHub stars, ships MCP server for AI workflows — D4Vinci · 2026-11-02
- Vercel open-sources Agent Desktop: reliable computer use via OS accessibility trees, 7.8k stars in a day — vercel-labs · 2026-11-02
- Karpathy's nanochat: the best ChatGPT $100 can buy, near 60k GitHub stars — karpathy · 2026-11-02