AI Engineer: Make Custom Evals a First-Class Citizen, Not Bigger Models
paulabartabajo_ · x · 2026-10-29
An AI lab practitioner argues that solving real business problems with AI isn't about the largest model or the newest agent framework—it's about making evals a first-class citizen and climbing them iteratively based on client feedback. Generic metrics are worse than useless: they create a false sense of progress.
His repeatable loop:
- Ship a first working version of the client's system
- Generate synthetic inputs and collect output traces
- Sample traces to identify main failure modes ("error categories")
- Build simple regex or LLM-as-a-judge evals measuring those failures
- Tune system parameters (prompts, tools, permissions, retrieval) and re-evaluate
- Return to step 3
When the set of error categories stops growing, your evals are right; climbing them brings the system closer to what the client actually needs.
More from coding & agent
- GitHub Copilot adds image generation in public preview, powered by GPT Image 2.5 Flare — caolonghua · 2026-10-29
- Anthropic surfaced 29,000+ Node.js bug candidates in six months; fixing is now the bottleneck — dscape · 2026-10-29
- Six months later nobody remembers why: Reddit debate on architecture decision records — djgoel123 · 2026-10-29
- Auto-Company: open-source firm of 14 AI agents runs 24/7 with one markdown file as memory — JafarNajafov · 2026-10-29
- One rule for posting agents: never auto-retry a failed publish — petrusenko_max · 2026-10-29
- Explosion ships agent-ready NLP project templates that cut pipeline cost 20x — _inesmontani · 2026-10-29