Agents on Rails Stage 3: GPT-6.1 Sol nears Opus 5.5 at one-thirteenth the cost ($0.80 vs $10.53)
npew · x · 2026-10-14
Rails' Agents on Rails benchmark returns with Stage 3, testing top models on 7 multi-step tasks spanning 3–8 tickets each, from building Rails apps from scratch to navigating messy legacy codebases.
- Best overall: OpenAI GPT-6.1 Sol — 60.8% success at $0.80 and 23m35s per run, the best accuracy/speed/cost balance
- Most accurate: Anthropic Opus 5.5 at max effort — 65.3%, but among the slowest and most expensive
- Fastest: OpenAI GPT-6 Astra at medium effort — 16m36s, 54% success
dhh highlighted that Sol finishes within 5 points of Opus at under a tenth of the cost, calling the value incredible.
More from coding & agent
- Google offers 10,000 scientists free 6-month top-tier Antigravity plans — danielrock · 2026-10-14
- CineRecon Benchmark: 12 Frontier Models Struggle to Reconstruct Editable 3D Scenes from Video — hkuhk · 2026-10-14
- Meituan's ClawLoop: Self-Improving Agents via Teacher-Free Harness-Native Training Loops — meituan-longcat · 2026-10-14
- Grok Bot's big leap: full breakdown of latest updates and autonomous business test — elonmusk · 2026-10-14
- Indie dev cuts homepage job list load from 1.8s to 0.1s in performance push — DanielLockyer · 2026-10-14
- Treat Your AI Harness Like Code: a CI Step That Ships Only Winning Candidates — AKAMA199 · 2026-10-14