A Small Frozen Eval Set Is Enough to Tell If Your Agent Actually Got Better
ChefOk4636 · reddit · 2026-11-14
A Reddit guide argues you don't need a public benchmark to know if your agent improved — a small frozen eval set suffices.
- Pick 20 tasks from the real workflow, each with input, allowed tools, and a clear definition of done
- Include tasks the agent should refuse or fail safely
- Freeze the set; rerun it after every prompt/tool/model/scaffolding change and keep traces
- Score few things that matter: real completion, unauthorized actions, knowing when to ask a human, time and money spent
If a change improves some tasks while quietly breaking others, you'll see it. If you can't write 20 realistic tasks, the agent isn't ready to run unattended. This complements — not replaces — sandboxing, secret handling, and budget caps.
More from coding & agent
- 4B specialized model hits 93.8% on bank support, beating proprietary models' 74% — MParakhin · 2026-11-14
- EnvOpt: Optimizing Agent Environments Beats Bigger Models on Cost and Pass Rate — AlexGDimakis · 2026-11-14
- workerd lands KV, Queues and Workflows in cluster, first release approaching — irvinebroque · 2026-11-14
- Researcher uses a coding agent to fetch latest Marin scaling-law fit parameters — RylanSchaeffer · 2026-11-14
- shadcn UI SDK lets agents edit forms through the same components as humans — shadcn · 2026-11-14
- Google Antigravity resets everyone's Gemini quota after a week of shipping — rseroter · 2026-11-14