Who grades the grader? A practical method for testing checks written by coding agents
jonah_omninode · reddit · 2026-09-29
The OmniNode author argues that a green build doesn't prove a check would catch the failure it was added to prevent. Key points:
- Bidirectional testing: validators must be tested with deliberately broken inputs (must fail) and their valid counterparts (must pass); testing failures alone lets a reject-everything validator look perfect.
- Keep regression pairs: for checks added after a real failure, keep a minimal reproduction of the defect and its fixed version, and confirm the check distinguishes them.
- He shares his inspection writeup and asks whether others use regression cases, mutation testing, or manual review.
More from coding & agent
- NetWasm compiles C# to a standalone 84 KB WebAssembly component with runtime and GC — shauntrennery · 2026-09-29
- OpenAI Engineer Details Six Prompt Caching Improvements Cutting Latency and Cost — pvncher · 2026-09-29
- Garry Tan Backs AI Browsers Like AsideAI: Agent Password Management Is the Key Layer — garrytan · 2026-09-29
- Meeting Prep Agent Separates App State in SQLite from Long-Term Memory — Datrika_Nandhini · 2026-09-29
- Chestnut unveils 18-DOF Aero Hand plus matching exoskeleton for zero-gap humanoid data capture — chris_j_paxton · 2026-09-29
- RubyLLM hits 1.1M downloads in a month, 13M total — kieranklaassen · 2026-09-29