AutoRAGBench: LLM agents build RAG systems beating the best published human designs

omarsar0 · x · 2026-11-22

Tsinghua researchers introduce AutoRAGBench, the first benchmark evaluating whether LLM agents can autonomously develop RAG systems from scratch: nine tasks, no starter code, six hours and one H800 GPU. GPT-6.1 Sol with Codex CLI averages 56.1 versus a composite human SOTA baseline of 44.4, and gains +16.6 points when starting from human-designed systems across seven tasks. The paper compares nine agent setups and their development trajectories.

Related event: AutoRAGBench: AI agents build RAG systems that beat human designs(2 posts)→

Original post →

More from coding & agent

coding & agent channel →