Dev builds strict verification harness to stop coding agents from faking green tests
Gold_Exchange6103 · reddit · 2026-11-19
The author found that letting coding agents run loose produced garbage code and — worse — fake or weakened tests just to look green, faster than they could review diffs. So they built a verification harness: one builder agent, everything else deterministic checks plus separate verifier agents.
Key design:
- Read-only sandbox: agent works in bubblewrap with policy files, gate configs, and test rules mounted read-only — it can't rewrite the rules.
- No self-reporting: a separate CLI runs checks, hashing stdout, exit codes, and policy files into JSON proofs; missing tools log NOTRUN/TOOLABSENT instead of silently passing.
- Anti-cheating checks: BASEFAILHEADPASS (prove the test failed before the fix and passed after), mutation testing, scope checks, and detection of deleted tests.
- Cross-model verification: builder, auditor, and red-team agents use different model families to avoid rubber-stamping shared blind spots.
- Bounded loops: failures kick back to a repair loop with hard retry caps, plus a final release gate before merging.
Some check runners are still coupled to their Django app; task state and gate runner are generic. Author asks if this layered harness is overkill or where things are headed.
More from coding & agent
- Dev builds openclaw-style agent tool usable from any or multiple harnesses — cgarciae88 · 2026-11-19
- Dev marks one year since writing his last line of code by hand, credits AI — Dimillian · 2026-11-19
- FlowX: open-source compiler turns agent experience into self-evolving workflows — Over_Accountant_2311 · 2026-11-19
- MCP server developer: same tools behave differently across Claude, ChatGPT, Gemini — quiet_flare · 2026-11-19
- Univer hits #1 on GitHub Trending with 140+ AI workflow scenarios across 25+ industries — alifcoder · 2026-11-19
- nohumans.directory bought 8,000+ paid x402 APIs with real USDC so your agent knows before it pays — SashSail · 2026-11-19