Graph+MCP Beats Traditional Markdown in Open-Source RAG Eval Experiment
JeremyCMorgan · x · 2026-07-31
This post shares an open-source knowledge base evaluation project comparing three ways to give LLMs microservice knowledge: Markdown, GraphRAG, and Graph+MCP.
The experiment evaluated completeness, scope, and determinism across 6 feature tasks using pairwise comparison:
- Markdown KB: Most readable but low completeness (0.65), weighted score 0.67.
- Graph-RAG: Moderate performance, improved completeness to 0.90, weighted score 0.74.
- Graph + MCP: Won comprehensively with a 0.97 weighted score, achieving perfect 1.0 completeness and exactness by computing impact via graph traversal.
The full repo with services and eval tasks is open source, encouraging developers to rerun it on their own architectures.
Related event: Graph+MCP Outperforms Traditional Markdown in Knowledge Base Evaluation(2 posts)→
More from coding & agent
- Flutter BLE DevTools: Debug Bluetooth Like Chrome DevTools — that_anokha_boy · 2026-07-31
- The 10x AI Coding Path: Building Loops for Agents to Self-Validate — brandon_galang · 2026-07-31
- Test Shows Qwen3.6 Outperforms Inkling-Small in Complex Code Generation and Self-Review — lilian_moraru · 2026-07-31
- AI Agents Reset 15 Years of Supply Chain Security: 9 of 11 MCP Markets Accept Malicious Code Unreviewed — ChuckDBrooks · 2026-07-31
- Test: Context Tree Architecture Helps Kimi K3 Outperform GPT-5.6 in Real Engineering Tasks — Still_Amphibian545 · 2026-07-31
- Multi-Claude Code Agents Autoformalize 500-Page Math Textbook at $100K Cost — gordic_aleksa · 2026-07-31