Model-written tests rejected a known-correct solution 77% of the time in agent pipeline test
deadatreides1 · reddit · 2026-10-05
A developer built the textbook coding-agent pipeline (contract → tests → code → run tests → repair), then fed every generated test suite a known-correct reference solution: 129 of 168 suites (77%) rejected it.
Key findings:
- The tests weren't lazy: average mutation score 0.965, and 81% of suites with a perfect 1.0 mutation score still failed the correct answer — thorough, confident, and testing the wrong spec.
- Favorite case: on a vowel-counting task, the tests and the bug shared the same misunderstanding (both expected 5 instead of 10 for mixed-case input), so the buggy code PASSED and repair never ran.
- Of 67 repair attempts, 50 were fixing code the tests had falsely rejected; only 1 of 17 real bugs was actually fixed.
- Better models false-reject less but still badly: llama-3.2-1b 0.92-1.0, qwen2.5-coder-1.5b 0.75, qwen3-1.7b 0.47-0.65.
- The whole pipeline didn't beat a dumb 4-8x sampling baseline at similar token budget.
Caveats: tiny local models (360M-1.7B), 6 tasks. The author's takeaway: the deciding test must come from the spec or human-written examples — a model may propose tests, but it doesn't get to be the judge.
Related event: Study Finds 77% of Model-Generated Tests Wrongly Reject Correct Code(2 posts)→
More from coding & agent
- TanStack AI adds experimental text-to-GLB 3D model and prompt-to-motion animation generation — bennash · 2026-11-20
- Dev builds typing survival game with troops and skill trees in ~24 hours using agents and kaigen3d — hullabaloo22 · 2026-11-20
- Harmonic's Aristotle writes proven-correct fusion plasma controllers in C and Verilog in 4 hours — satnam6502 · 2026-11-20
- R3 Code: A Native Rust Mac Client That Runs All Your Coding Agents — jasonkneen · 2026-11-20
- zlib: 2.4K-star open-source CLI turns Z-Library into a skill for AI agents — udmrzn · 2026-11-20
- Andriy Burkov's True Positive Weekly #186: what to know when agents do the work — burkov · 2026-11-20