24,888-repo benchmark finds zero AI agent violations of an inferred import rule
LogFun4751 · reddit · 2026-09-13
The author built a benchmark to test the popular claim that AI coding agents drift from prose rules (CLAUDE.md/AGENTS.md) and need deterministic lint enforcement. He mined an import-boundary rule from a near-census of 24,888 public TypeScript repos (entry files must not import UI components), then ran Claude, GPT and Gemini through a realistic coding task under four conditions, from unguarded control to hard lint error, plus a longer task deliberately nudging toward violation with weaker, cheaper models. Result: zero violations across every model and condition — even with no rules at all. He publishes the null result with pre-registration, harness and raw data, acknowledging the narrow scope (one rule, one repo) and inviting others to break an inferred import boundary.
More from coding & agent
- GitHub Copilot app is building richer dashboard widgets for personalized data views — mariorod1 · 2026-10-31
- Todoist's self-healing agent auto-fixes 1 in 5 Sentry issues and opens PRs — zeeg · 2026-10-31
- Dev's Multica 1.0 build log: 432M tokens, 89 agent runs, just $183 — jiayuan_jy · 2026-10-31
- Stardock's QA now runs agents that play the game and file reports — draginol · 2026-10-31
- Limma MCP structures unstructured data end-to-end so agents never miss a detail — FlimsyIngenuity7248 · 2026-10-31
- Agent audit metric: can you name which tests ran and files changed without reading its summary? — ronbodkin · 2026-10-31