Small Models Write the Tests Too: 77% of Self-Written Test Suites Rejected Correct Code
deadatreides1 · reddit · 2026-10-05
A Reddit experiment with four tiny GGUF models (360M–1.7B, 785 calls, 6 tasks) found 129 of 168 self-written test suites (77%) rejected a known-correct reference solution, despite a 0.965 average mutation score — thorough but testing the wrong spec. Of 67 repair attempts, 50 patched already-correct code. Plain resampling (pass@8 = 0.958) beat the pipeline at similar token cost. The author's fix: judges come from human-written specs, not model-proposed tests.
Related event: Study Finds 77% of Model-Generated Tests Wrongly Reject Correct Code(2 posts)→
More from coding & agent
- Nous Research partners with Microsoft to bring Hermes Agent into Teams and Azure — NousResearch · 2026-11-20
- We Listed Our MCP Server on 53 Directories: Only 16 Went Live in 11 Days — B3N0U · 2026-11-20
- Replit Missions crawls 1,000+ e-sign flows with browser use, rebuilds them 1:1 — amasad · 2026-11-20
- Replit clones a leading AI assistant in 3 days with Missions, open-sources self-hosted agent Amuse — amasad · 2026-11-20
- Replit CEO: 'having a contract with your agent' is a paradigm many will copy — amasad · 2026-11-20
- Google launches Tool Retrieval in Gemini API preview to scale agents with hundreds of tools — OnlineInference · 2026-11-20