Apple releases RSI-Inference Bench: best agent solves just 26.7% of vLLM/SGLang tasks
SumitGup · x · 2026-10-17
Apple ML Research introduces RSI-Inference Bench, asking whether AI agents can improve the systems that run AI. The team spent $100K in API tokens and 50K GPU hours building environments from real-world vLLM and SGLang tasks covering the full inference engineering lifecycle: Day-0 model support, inference optimization, and bug fixes. The benchmark is unsaturated — the best agent reaches only 26.7% pass@1 — and offers verifiable binary and continuous rewards. Website and code are available.
More from coding & agent
- Garry Tan shows agents coding his new project and handling DevOps in one go — garrytan · 2026-10-17
- Rod Johnson demos Embabel Worlds: vibe coding safely over knowledge graphs — springrod · 2026-10-17
- React core team member open-sources pstack, a skills stack for trustworthy AI coding agents — alexcovo_eth · 2026-10-17
- Handing a project to a new Claude? Make it take a 13-part onboarding course — ivan_bezdomny · 2026-10-17
- Musician makes hand-drawn 2D music video entirely in code with Claude and Remotion — pizzaisprettyneato · 2026-10-17
- How Codex Happened: Dan Shipper's 8,000-word AI-assisted feature for Every — lmoroney · 2026-10-17