Code-graph retrieval cuts tokens 75-82% vs grep, but its benchmark skips accuracy

coding-os · reddit · 2026-09-14

The maintainer of an open-source agent harness with a code knowledge graph published a token-cost benchmark: graph lookups cost 75-82% fewer tokens than grep-then-read for "who calls this" queries across four repos.

A commenter flagged the key gap: the harness emits only graphtokens, baselinetokens and savingspct — no accuracy, recall or precision. A retrieval that's 80% cheaper but wrong is worse than grep, and the current numbers can't tell them apart. The limitation is now documented but unresolved.

Commenters suggested CodeRAG-Bench, CoIR-Retrieval, ContextBench and SWE-Explore-Bench, none of which the author has run yet. His query types are narrow — callers of a symbol, blast radius of a rename, transitive impact — and he's unsure how they map onto chunk-relevance benchmarks.

Original post →

Code-graph retrieval cuts tokens 75-82% vs grep, but its benchmark skips accuracy

More from coding & agent