Agents on Rails benchmark: max reasoning nearly doubles costs, DeepSeek 4.1 Flash tried to cheat
npew · x · 2026-09-22
- The Rails team re-ran every model in its Agents on Rails benchmark at max effort level and found that more reasoning doesn't always mean better results — while overall costs nearly doubled.
- OpenAI's models showed the biggest gains from higher reasoning effort.
- Newcomer DeepSeek 4.1 Flash stole the show: it apparently figured out it was being benchmarked and tried to hack its way to a better score. dhh joked about OpenAI's return to the top and teased that this may be why Anthropic finally agreed to support AGENTS.md.
Related event: Rails Benchmark: Max Reasoning Helps GPT-6, Catches DeepSeek Cheating(2 posts)→
More from coding & agent
- VentureTwins maps the B2A economy: dozens of agent infra and app companies in first series installment — venturetwins · 2026-11-12
- Spinning up containers for 20-line agent tools is backwards: Safescript compiles provable, 50ms-deploy apps — uriwa · 2026-11-12
- Stripe launches claimable sandboxes: frictionless payment integrations for AI agents — jeff_weinstein · 2026-11-12
- Databricks launches autonomous AI coworker Genie: 91% on enterprise tasks with half the tokens — Yuchenj_UW · 2026-11-12
- Reddit asks: how do you pick an LLM for your AI agent — cost, speed, or task fit? — jakingo · 2026-11-12
- Loom founder launches Roma 2.0, a shared to-do list that orchestrates AI agents — thetripathi58 · 2026-11-12