Same Model Benchmark: Agent A (45/50) vs Agent B (43/50) with Similar Cost
donk8r · reddit · 2026-08-22
A benchmark test compared two open-source coding agents (Octomind vs. a competitor) using identical settings (deepseek-v4-flash, same prompts, no web access) across 50 tasks. Results show Octomind scored 45/50 vs. 43/50, with judge scores of 88.6 vs. 85.6, and near-identical costs ($1.59 vs. $1.53). Although Octomind had a slower mean time, its median was faster (5.5m vs 7.0m), winning in 31 of 50 cases. The mean time discrepancy was caused by a single extreme failure case, revealing a bug in Octomind's no-progress detection.
More from coding & agent
- Vercel Agent integrates with Slack Code for in-channel deployment — kmiddleton14 · 2026-08-22
- Grok Bot v0.23.0 adds Channels: enabling concurrent multi-project management — altryne · 2026-08-22
- Ask Claude to spawn a new session when it spots new work mid-session — EricBuess · 2026-08-22
- Study finds deep research agents have selective memory: focusing on start and end — yuz9yuz · 2026-08-22
- New AI Coding Crash Course targets team training to regain code control — mattpocockuk · 2026-08-22
- New ComfyUI Node: CLIP-Only LoRA Loader Supports Pure LLM Workflows — Quagmire94 · 2026-08-22