Reliable Graph-RAG for Codebases: AST-Derived Graphs vs LLM-Extracted Knowledge Graphs
Manideep Reddy Chinthareddy
cs.SE, cs.AI
2026-01-14
On three Java repos, a Tree-sitter graph answers 15/15 Shopizer architecture questions. An LLM-extracted graph skips 377 of 1210 files, costs about 20× more, and scores 13/15.
Questions such as "which controllers use the shopping-cart logic" are multi-hop. The evidence sits on injection chains, interface wiring, and inheritance, not in the chunks that look most similar to the query. Vector RAG often retrieves the implementation class, drops the controllers that consume it, and the generator fills the gap with framework folklore.
GraphRAG tries to restore those hops. The graph can come from an LLM that emits entities and dependencies per file, or from an AST that extracts types and edges deterministically. The first path sounds more semantic. It can also silently drop files at index time.
Three pipelines share the same 15 architecture and tracing questions on Shopizer, ThingsBoard, and OpenMRS Core, all Java.
Answers are labeled correct, partial, or incorrect by whether they name real entities and relations and avoid invented components. Coverage is measured from scanned files, embedded chunks, graph size, and skip logs.
On Shopizer, DKB is 15/15, LLM-KB 13/15 with two partials, and No-Graph 6/15 with four partials and five incorrect, the last of these hallucinating most on architecture-discovery questions. Across 45 questions: DKB 43/2/0, LLM-KB 38/5/2, No-Graph 31/9/5. On ThingsBoard the vector baseline is already strong at 14/15, DKB ties it, and LLM-KB falls to 12/15.
Indexing reliability is the sharper gap. All three pipelines discover 1210 Java files on Shopizer. LLM-KB skips 377 (file success 0.688), embeds 3465 chunks (0.641 coverage versus No-Graph), and builds 842 nodes. DKB embeds 4873 chunks (0.902) and 1158 nodes. File success on ThingsBoard and OpenMRS is 0.806 and 0.650 for LLM-KB, with chunk coverage 0.706 and 0.633; DKB stays near 0.99.
Graph build is 2.81 s for DKB versus 200.14 s for LLM-KB on Shopizer. End-to-end cost (index plus 15 questions) is $0.04 / $0.09 / $0.79 on Shopizer and $0.149 / $0.317 / $6.80 on OpenMRS+ThingsBoard, about 46× for LLM-KB versus No-Graph. Query latency sits in the 10–15 s band for all three; LLM-KB has heavier tails.
Do not let an LLM build the code knowledge graph at index time. Injection, inheritance, and implements edges are compiler-visible; an AST lays them down in seconds with full file coverage. LLM extraction drops files under batching and schema constraints, and those files never enter embeddings or neighborhood expansion.
Vector RAG is fine for local questions. Once the task is "find upstream consumers through an interface," bidirectional expansion plus InterfaceConsumerExpand is the actual delta. The cost is a narrow edge schema: reflection and runtime-generated behavior are invisible.
All three repos are Java. AST extraction misses reflection, runtime codegen, and purely dynamic dispatch. DKB chunk coverage on Shopizer is 0.902 because embedding is coupled to discovering a top-level type, so typeless utility files can drop out. Correctness is single-annotator per question, with no agreement study. LLM-KB's denser dependson edges are not comparable to DKB's typed injects/extends/implements counts.