Benchmarking two code-graph tools on 20 real pydantic commits: precision tops out at 34.5%
Specialist-Can2889 · reddit · 2026-10-30
The author benchmarked two open-source code-graph tools on 20 real pydantic commits, asking "which files need to change when this function changes?" Method frozen beforehand: index the repo at each commit's parent, query callers of every changed function, score against files actually changed together.
Results
- EngramGraph (the author's own MIT project): 34.5% precision
- code-review-graph (32k stars): 5% with bare function names, declining 18 of 42 lookups as ambiguous due to test-name collisions; 15% when re-queried with fully qualified names
- code-review-graph clearly wins on indexing speed: 16.6s vs 455s median
Limits: one repo, 20 commits, direct callers only; many co-changed files aren't call relations, so both scores are low; code-review-graph's headline diff-based impact analysis wasn't compared. The author discloses the conflict of interest and notes text/summary image were AI-drafted, with numbers personally verified.
More from coding & agent
- Autonomous agents debated his sleep rule overnight, then called him at 7am to save a release — airesearch12 · 2026-10-30
- Dev's slangy commit message 'api key ui very nice' becomes a self-own screenshot — itsOmSarraf_ · 2026-10-30
- lattedit: a local-first browser LaTeX editor with a JSON API for AI agents — cneuralnetwork · 2026-10-30
- Pedro Domingos: COBOL will outlive Claude Code — pmddomingos · 2026-10-30
- Anthropic's Felix Rieseberg: 4 ways to use AI more ambitiously — speed, altitude, depth, range — FinanceYF5 · 2026-10-30
- kalomaze: agents keep sneaking MSE regression losses into multimodally conditional ML problems — kalomaze · 2026-10-30