27B open-source model beats GPT-5.5 and Opus 4.8 on desktop tasks at 1.5% of the cost

alex_verem · x · 2026-10-23

Agent S4: Evolve the tools, not the model

Researchers at Simular observed that most computer-use agents operate through interfaces designed for humans — screenshot, click, type, screenshot again — which wastes enormous effort (e.g., filling a spreadsheet cell by cell instead of running one line of code). Their key insight: the bottleneck may sit in the tools we hand agents, not the model itself.

How it works

Agent S4 is a second AI that watches the first agent work, identifies where it wastes effort, and writes better tools for it. It then selects the best-performing version on held-out test tasks.

OSWorld V2 results (108 real desktop tasks)

Open-source Qwen3.8-27B: full-task completion jumped from 5.6% to 21.3%
GPT-5.5: 13.0%; Claude Opus 4.8: 20.6%
Cost per task: $1.10 for the 27B model vs. $25.50 (GPT-5.5) and $72.40 (Opus 4.8)

Concrete effects

Expense-receipt task: with human-designed tools the agent opened emails one by one, dismissed five download warnings, and scrolled a card statement eight times; with evolved tools it read the mailbox and charges as text and downloaded three PDFs from links.
On ScienceBoard (astronomy, molecules, maps, math-proof apps): evolved tools hit 58.6% vs. 41.4% for the standard setup.

Caveats

The $1.10/task excludes tool-building cost, which took 30 rounds of evolution. The paper's broader question: how much of the agent race is being spent on the wrong part of the problem? Code is open source.

Original post →

27B open-source model beats GPT-5.5 and Opus 4.8 on desktop tasks at 1.5% of the cost

More from coding & agent