360-trial benchmark finds code knowledge graphs give coding agents no clear accuracy gain
JeremyCMorgan · x · 2026-11-10
A benchmark testing graphify — which turns codebases into call/import graphs — asks whether graph facts actually help coding agents. Running 360 trials (12 questions, 3 repos) with claude code (sonnet 5.5) and codex (gpt-6.1-sol), the test was prompted by a user report in graphify issue #985 that plain claude code gave similar answers with fewer tokens.
Key findings
- Graph facts, delivered via a purpose-built hook prototype or two graph tools, did not make strong agents clearly more accurate or cheaper. Agents found answer files with grep within the first few calls, then still read the files to verify — graph notes replaced none of that reading.
- A revised hook now cuts tokens by 13% versus plain grep at roughly the same accuracy, though this comes from a small run.
The author suggests reading this before adding graph infrastructure to a coding stack, and separately lists six guesses about where graphs might genuinely help, with evidence.
More from coding & agent
- Fable 5.2 models a Bugatti Tourbillon in Blender from a single prompt in 2.5 hours — socialwithaayan · 2026-11-10
- Pake: Rust-based tool packs any webpage into a lightweight desktop app, 539K downloads — Aiden_Tech_Ai · 2026-11-10
- Frontier3DBench: 145 tasks test whether coding agents can replace 3D vision pipelines — Aiden_Tech_Ai · 2026-11-10
- Human-in-the-loop is easy; keeping the human attentive is hard — 4 metrics to test if your AI approvals mean anything — alex_verem · 2026-11-10
- Paid agents vs open-source clones: self-hosting shifts machines, model bills and data to you — femke_plantinga · 2026-11-10
- Your agent isn't dumb, it's just multiplying: 95% per-step reliability drops to 36% over 20 steps — Infinite_Web1576 · 2026-11-10