Treat Your AI Harness Like Code: a CI Step That Ships Only Winning Candidates
AKAMA199 · reddit · 2026-10-14
A practitioner shares how to keep a local-model eval harness from rotting by turning it into CI: freeze 10-20 task scenarios asserting on decisions not wording, treat every rules/prompt change as a candidate that must beat the current tree on the same model, let the runtime write per-inference receipts (reef's x-reef-agent-record-id header), and only publish a winning tree as a release — rollback is just checking out the last passing version. No GPU needed; the harness is just files. The step people skip, he notes, is making the suite actually decide what ships.
More from coding & agent
- ArkForge Trust Layer signs AI agent HTTP calls with Ed25519 and Rekor anchoring — modelcontextprotocol · 2026-10-14
- AI agent stumped by 'press & hold' bot check, forced to ask human for help — xeophon · 2026-10-14
- Agent runs all day, delivers 150 lines of code, gets roasted as 'worse than humans' — lxfater · 2026-10-14
- Pydantic AI adds long-decode support for Gemini 4 Argon, auto-following continuation tokens past the 32K limit — m4rkmc · 2026-10-14
- A Full QC Pipeline for RL Data: Intake Review, Active Testing, Packaging — joecole · 2026-10-14
- Google offers 10,000 scientists free 6-month top-tier Antigravity plans — danielrock · 2026-10-14