Evals are the definition of done: 'Demo-ready' isn't 'monetize-ready' for AI agents
leveragedupside · x · 2026-10-20
The author's project agent declared his product "done," the content agent drafted launch posts, but the QA agent—checking from a paying customer's perspective—found missing competitor links, wrong file paths, and a skipped comparison sheet. Verdict: "Demo-ready. Not monetize-ready." Nobody lied; they used two different definitions of done.
Core argument: managers solved this decades ago with MBO/OKR and Definition of Done—evals are the definition of done for AI.
Two builder cases (their claims, not author's tests):
- A WhatsApp lead-reply bot broke on real leads (30-word answers, wrong voice-note language, chatting with non-buyers); now at v4.7, each break became an eval—one sentence, one question, language by country code, silence on non-leads.
- An HVAC booking agent: "booked" isn't done—right job type, right tech, buffers and travel time checked before offering a slot.
His four-step method: 1) define done before running; 2) check against what the customer pays for; 3) turn every real failure into a new check; 4) once checks are solid, stop gating everything and watch the data. Good evals stop the human being the bottleneck.
More from coding & agent
- Notion AI ships major upgrade: smarter Agents, Auto 90% cheaper, BYOK support — ivanhzhao · 2026-10-20
- iPhone Action Button + Grok voice + HeyGen MCP turns a sentence into an avatar video — HeyGen · 2026-10-20
- Vibe-coded game League of Spawn adds Arenas: 15-minute MOBA team fights — majidmanzarpour · 2026-10-20
- esp32sim-mcp: MCP server lets AI agents build and test ESP32 firmware in a local emulator — EmanueleStrazzullo · 2026-10-20
- GameSummoner one-line local install: prompt-to-game pipeline runs fully local on a 5090 — demonicpigg · 2026-10-20
- Heavy Grok Bot user argues personal agents must stay open-source, shares backup SOP — WolframRvnwlf · 2026-10-20