AutoRAGBench: LLM Agents Build RAG Systems From Scratch, GPT-6.1 Scores 56.1 Beating Human SOTA
_reachsumit · x · 2026-11-20
AutoRAGBench is the first benchmark evaluating how well LLM agents autonomously develop RAG systems from scratch in an open code space.
- Nine RAG development tasks, no starter code or prescribed strategies, 6-hour budget on a single H800; agents analyze data, implement pipelines, and iterate on experimental feedback
- Nine agent configurations evaluated: GPT-6.1 Sol with Codex CLI leads at 56.1 average score, beating the composite Human-SOTA baseline of 44.4
- Results show agents can automate effective RAG system development; the paper also analyzes development trajectories
More from coding & agent
- OtterKit turns local MCP servers into public URLs with one command for Claude web testing — Conscious-Return-183 · 2026-11-20
- Andrew Carr blocks Antigravity's mobile PWA — agentic coding in bed is too addictive — andrew_n_carr · 2026-11-20
- Meta's Agentic Feature Optimization Lifts Ad Ranking Models 0.20% in Online A/B Tests — _reachsumit · 2026-11-20
- Procedural Gojira Built with Gemini and Three.js TSL — majidmanzarpour · 2026-11-20
- User finds their AI agent quietly ate 54GB of disk with unused git worktrees — itsOmSarraf_ · 2026-11-20
- Harness engineering gap: Provider harness 39.0% vs Mini-SWE 12.8% on same model — andrew_n_carr · 2026-11-20