24,888-repo benchmark finds zero AI agent violations of an inferred import rule
LogFun4751 · reddit · 2026-09-13
The author built a benchmark to test the popular claim that AI coding agents drift from prose rules (CLAUDE.md/AGENTS.md) and need deterministic lint enforcement. He mined an import-boundary rule from a near-census of 24,888 public TypeScript repos (entry files must not import UI components), then ran Claude, GPT and Gemini through a realistic coding task under four conditions, from unguarded control to hard lint error, plus a longer task deliberately nudging toward violation with weaker, cheaper models. Result: zero violations across every model and condition — even with no rules at all. He publishes the null result with pre-registration, harness and raw data, acknowledging the narrow scope (one rule, one repo) and inviting others to break an inferred import boundary.
More from coding & agent
- Dev lets AI bots run wild in a virtual world, plans 30,000-bit rewards — Daniel_Farinax · 2026-09-13
- Nous Research's Hermes Agent PR Adds Per-Bot Xfce Desktops You Can Take Over — Teknium · 2026-09-13
- "Vibe coding" is now redundant—it's just coding, and it forces you into systems thinking — SimonBalmain · 2026-09-13
- Developer jokes: let Claude Code and Codex review each other, I just say LGTM — prateekj · 2026-09-13
- Astra builds a tiny planet with 7 islands — and a browser airline to run — TheMoonMidas · 2026-09-13
- SlopCodeBench draws praise as a new benchmark for multi-turn code degradation — cedric_chee · 2026-09-13