omarsar0: Without an 'eval factory' there's no 'software factory' — agent automation is worthless without evals
omarsar0 · x · 2026-11-15
AI commentator Elvis Saravia (omarsar0) distilled a consensus from conversations with agent builders: without evals, the recursive self-improving loop is just more lines of code with no clear value.
Key points:
- "You can't effectively automate what you cannot properly measure"
- Any product used for agent automation should treat building and measuring evals as a first-class capability, or at least support it easily
- This applies across all functions of an organization, not just software engineering
His one-liner: "There is no 'software factory' without an 'eval factory'."
More from coding & agent
- MaxxBench: tiny task tweaks spread coding-agent scores by 23 points — MFarajtabar · 2026-11-15
- Managing multiple Codex sessions is breaking down, Reddit user ventshelp — Excellent-Wheel7769 · 2026-11-15
- DSPy tunes prompts and weights to boost LLM performance, says practitioner — mdancho84 · 2026-11-15
- Stop prompting, start programming LLMs: an intro to Stanford's DSPy — mdancho84 · 2026-11-15
- Security experts warn: many AI agents run headless Chrome with --no-sandbox — matthew_d_green · 2026-11-15
- Ex-Microsoft CTO Parakhin's 48h AI usage: Codex/Astra at ~50% of tokens — MParakhin · 2026-11-15