AutoRAGBench: A Benchmark for LLM Agents Automating RAG System Development
Hang Zhang, Yuzhi Pan, Ziyao Zhong, Chaokun Wang, Shuo Cao, Zeyu Huang, Yue Xue, Fang Niu, Bofan Xie, Guanchen Ge, Zecheng Li, Leqi Zheng, Ziyang Liu, Wei Wang, Miao Gu
cs.CL, cs.AI, cs.IR
2026-11-19
On AutoRAGBench, LLM agents build RAG systems from scratch in six hours on one H800 GPU; GPT-6.1 Sol averages 56.1, 11.7 points above the composite Human-SOTA baseline.
Building a good RAG system is craft work. How you chunk the corpus, build the index, retrieve, and select evidence all depend on the task: multi-hop questions need evidence combined across documents, knowledge-graph questions need entity linking and graph traversal. No configuration wins everywhere, so developers loop through analysis, implementation, testing, and revision.
Neither existing line of work answers whether agents can do this job. RAG evaluation benchmarks (RAGBench, CRAG) measure components and answer quality, not the act of building the system. Automated optimizers such as AutoRAG, FlashRAG, and HAMMER search within predefined component and hyperparameter spaces. AutoRAGBench, from Tsinghua's School of Software with Weibo, reframes the question: hand an agent a corpus, QA examples, one H800 GPU, and six hours. No starter code, no prescribed design space. Write, test, and refine the system the way an engineer would.
The design targets two problems: diverse tasks in clean environments, and evaluation of the iteration process rather than just the artifact.
Nine tasks in five categories cover open-domain QA (NQ), document multi-hop QA (HotpotQA, MuSiQue, MultiHop-RAG), domain QA (PubMedQA, QASPER), fact verification (FEVER), and knowledge-graph QA (WebQSP, CWQ). Corpora range from 609 news articles to 21 million Wikipedia passages; each task keeps 1,000 hidden test questions.
Every build runs in a fresh isolated container, with the test set outside. Systems may retrieve only from the supplied corpus; downloading external datasets is banned. A DeepSeek model audits full development trajectories for violations, backed by manual review; violating runs are rerun or scored at the BM25 baseline.
The key evaluation choice decouples the retriever from the reader. Five held-out 3B-4B answering models (Gemma-3-4B, Phi-4-mini, and others) each answer every test question from the same retrieved evidence, and scores are averaged. Reader differences wash out, so the score mostly reflects evidence quality. Submitted systems expose a POST /retrieve endpoint with per-question budgets of four auxiliary LLM calls and 60 seconds; generated summaries may guide retrieval but cannot replace corpus evidence.
On the process side, trajectories are segmented into prepare, code, run-test, and analyze activities, and each build's development scores are tracked from first version to last and against the peak.
| Method | Nine-task average |
| GPT-6.1 Sol (Codex CLI) | 56.1 |
| GPT-6 Sol (Codex CLI) | 54.1 |
| GLM-5.3 (Claude Code) | 50.5 |
| Reference RAG (hybrid + rerank) | 46.4 |
| Human-SOTA (composite) | 44.4 |
| Naive BM25 RAG | 36.7 |
| Closed-book | 25.3 |
All nine agent configurations beat BM25, seven beat Reference RAG, and all nine sit above the Human-SOTA composite. The leader's 9.7-point margin over Reference RAG equals the gap between BM25 and Reference RAG itself.
Gains concentrate on knowledge-graph tasks: on WebQSP all nine configurations beat Reference RAG by 14.1 points on average, and the best CWQ gain is 27.3. Strategy counts explain it: all 18 knowledge-graph systems implement graph traversal, an option absent from both automated frameworks' search spaces. HotpotQA is the opposite case; the best configuration gains 13.0 while the mean gain across agents is 0.6.
Speed is the cost. Agent systems average 0.50 to 2.01 seconds per question against Reference RAG's 0.35, with GPT-6.1 Sol at 1.23. Extra time buys no guarantee: DeepSeek V4.1 Flash takes the longest (2.01 s) and scores below five configurations.
Iteration works, with traps. Of 77 builds with comparable development scores, 72 improved on their first score by at least a point at some stage; median first-to-last gain is 5.5, and median time to peak is hour four. Twenty builds ended at least a point below their peak. GPT-5.6 Luna shows the overfitting risk: the largest median development gain (10.0) but tenth place on hidden tests, with 85% of its self-checks on one repeatedly used question file, scoring 53.0 on questions it had tested against 35.0 on new ones.
Two extensions matter for practitioners. Given existing human systems on seven tasks, GPT-6.1 Sol improves them by 16.6 points on average within six hours (range 7.4 to 26.4); HyDE on HotpotQA goes from 37.37 to 63.74. Against automated frameworks on six common tasks, the agent averages 57.0 versus Agentic AutoRAG's 44.5 and FlashRAG's 43.6, and FlashRAG fails to finish indexing HotpotQA within six hours. Total API cost for nine tasks runs about $167 for GPT-6.1 Sol and $9.5 for DeepSeek V4.1 Flash. Testing and analysis consume 57-87% of build time, at a median of 48-68 modify-and-evaluate rounds per build.
This moves the "can agents do real engineering" question from writing a function to shipping a system, and the answer is yes at a price many teams would pay: six hours and about $167 with no starter code.
The most immediately usable result is the improvement experiment. Most teams already have a working RAG system; letting an agent revise it for six hours returned 16.6 points on average here. The knowledge-graph results also make a substantive point against config-search frameworks: open code space produces designs (graph traversal) their search spaces cannot reach.
Keep the score structure in perspective. Human-SOTA is a per-task envelope over adapted implementations with a 24-hour budget, not a deployable system, so the 11.7-point headline overstates the match-up. What the benchmark establishes is that the agent engineering loop works, not that RAG research is done.
Stated in the paper: strategy labels for the 81 final systems were coded by one person with no second-coder verification; activity-labeling rules were developed with model help and lack independent human validation; Human-SOTA scores come from adapted implementations that need not reproduce original papers; development scores are within-build descriptive measures over differing question sets.
Questions remain after reading. All five answering models are 3B-4B, so absolute scores depend on that choice; rank stability was checked, stronger readers were not. A repeated CWQ run fell from 34.58 to 33.20, so single-run noise puts sub-point average gaps in doubt. Twenty of 77 builds ended below their peak, yet checkpoint selection at the six-hour freeze is not separately quantified. The framework comparison's adapter omits some diagnostics, so how much of that gap comes from limited search spaces is not clean.