6 new October models, same 5 coding tasks: all pass but costs range $0.02 to $0.65
latent_lurker8587 · reddit · 2026-10-31
The author tested six October model releases on 5 small coding tasks with hidden traps (YAML override, stats script on hidden data, slugify fix, ERROR-line counting, look-alike rename), using Atomic Agent 0.6.13 in cloud auto-approve mode via OpenRouter, one run each with a 10-minute cap.
Results (cost for all 5 tasks):
- DeepSeek V4.1 Flash: 5/5, median 14s, $0.02
- Claude Haiku 5.5: 5/5, 12s, $0.035 (fastest)
- Gemini 4 Argon: 5/5, 43s, $0.47
- Kimi K3.1 Pro: 5/5, 40s, $0.59
- GPT-6.1 Terra: 5/5, 18s, $0.65
- Recursive Flint (free tier): 1/5, despite a vendor-claimed 74.2 SWE-bench Verified
Takeaways:
- All major models passed everything, but prices differ 30x for identical passes — and DeepSeek was faster. Haiku 5.5 is the quiet winner.
- Flint underperformed badly, though its free endpoint kept 503-ing all afternoon ('no upstreams available'), so the run may be unfair; a rerun via a second agent never got a clean test.
- Practical lessons: keeping the agent's prompt prefix identical per step hit 79–83% prompt-cache rates, the main reason even pricey models stayed under $1; per-step logging revealed Flint was stuck, not just slow.
Small sample — a first look, not a final ranking.
Related event: Six October AI models pass same coding tests, but costs differ 30x(2 posts)→
More from coding & agent
- Playwright MCP Server lets AI assistants drive remote browsers — modelcontextprotocol · 2026-10-31
- zero-core-tools: MCP connector bundling scraping, code review, sentiment — modelcontextprotocol · 2026-10-31
- Codex power user shows 71.5B lifetime tokens and a 262-day daily streak — TheMoonMidas · 2026-10-31
- ComfyUI-Dials: prompt pairs become 14KB sliders controlling video motion & mood, no LoRA training — MatlowAI · 2026-10-31
- Dev Asks GPT-6 Astra to Motorize an Office Chair, Then Builds It in Real Life — sdand · 2026-10-31
- Truo: read-only MCP server pipes live Australian bank data into AI assistants — truo_au · 2026-10-31