AutoRAGBench: LLM agents build RAG systems beating the best published human designs
omarsar0 · x · 2026-11-22
Tsinghua researchers introduce AutoRAGBench, the first benchmark evaluating whether LLM agents can autonomously develop RAG systems from scratch: nine tasks, no starter code, six hours and one H800 GPU. GPT-6.1 Sol with Codex CLI averages 56.1 versus a composite human SOTA baseline of 44.4, and gains +16.6 points when starting from human-designed systems across seven tasks. The paper compares nine agent setups and their development trajectories.
Related event: AutoRAGBench: AI agents build RAG systems that beat human designs(2 posts)→
More from coding & agent
- If an agent can still move money after its prompt says no, the boundary is in the wrong place — ExecLayer_io · 2026-11-22
- Stop @-ing your agent in Slack — let Jev decide when it should respond — DavidWells · 2026-11-22
- Intent adds notch launcher, Claude Code and Codex session import — Wattenberger · 2026-11-22
- rakyll explains AX: a low-opinion job orchestrator that agents control themselves — rakyll · 2026-11-22
- Agent builds a full Shopify storefront with zero manual edits under a code grammar — FrBatista · 2026-11-22
- Gemini Neon builds an interactive USL lock-contention UI exposing a testing blind spot — tianyin_xu · 2026-11-22