Your agent eval gate passed, but the next served request still scored zero
papaparija25 · reddit · 2026-10-17
The author recounts an unattended agent run that exited 0 with a green dashboard for a week while outputting a hallucinated no-op—hence "check the artefact, not the exit code." But he found a case where artefact checking still wasn't enough. Using reef's harness-evolution tutorial (three exact-answer coding tasks: sieve, fib, csv; an edit publishes only when a previously failing task passes), he shows: models that already pass everything record no failures so evolve never runs; unsupported model names record "skipped: no proposal"—no-ops written down as no-ops. Yet in the self-tools run, a 7B proposed two rules entries, the gate published them, and a later served request on sieve still scored 0.0. The takeaway: a gate is a comparison on the task list you gave it and says nothing about the request that arrives after; keep scoring served requests too—the gate's yes and the next request's no are two different measurements, and you want both.
More from coding & agent
- Dev swaps codex remote and Claude remote control for T3 Code mobile — rudrank · 2026-10-17
- Perplexity Flash matches Wald's accuracy at nearly 2x speed in local agent-decision benchmark — WolframRvnwlf · 2026-10-17
- Celesto CI runs cache-hit jobs 23% faster than GitHub Actions on half the CPUs — aniketmaurya · 2026-10-17
- Jev vs tabular foundation models: zero-shot decision routing is not grounded prediction — Arindam_1729 · 2026-10-17
- BFL Updates FLUX 3 Image Klein Workflows: Prompt Upsampling and 2K Resolution Required — bfl_ai · 2026-10-17
- Claude can now reverse-engineer and rebuild apps like TradingView and OBS without source code — PrajwalTomar_ · 2026-10-17