Spidermonkey trims 10% agent steps on weak models, no gain on strong ones

mathemagic1an · x · 2026-10-14

Testing of Spidermonkey (a System 1 setup: tests watcher + Jev decision model) on SWE-bench shows GPT-6 Luna saves 10% steps at the same 18/42 solve rate across 42 replicated attempts, while GPT-6.1 Sol sees no gain and runs 21% slower. The real value is in chaotic live environments: hung test runs end in 15s instead of 90s (-54%) with no false interrupts. Traditional isolated-task evals like SWE-bench miss this benefit, and as coding agents scale, delegating environment monitoring to the 'subconscious' becomes essential.

Original post →

Spidermonkey trims 10% agent steps on weak models, no gain on strong ones

More from coding & agent