I benchmarked 10 AI agents on a 4B local LLM: seven aced it, Codex CLI went 0/5
GapNew4766 · reddit · 2026-10-21
A Reddit user ran 10 coding agents against Qwen3.5-4B (Q4KM) with isolated config folders, headless auto-confirmed runs, 5 fixed tasks (YAML port extraction, script writing, hidden-case slugify bugfix, log error counting, multi-file rename), each scored by an automated check script.
Results: Pi v1.3.0 swept 5/5 fastest at 1m36s; goose, Crush, Atomic Agent, Cline and OpenClaw also went 5/5. OpenCode passed everything but took 17m43s, badly overthinking simple steps. Hermes failed the hidden slugify case (c++-&-rust-2026 vs c-rust-2026); Qwen Code miscounted 43 ERROR lines as 40. Codex CLI went 0/5 without a single request reaching the model — v0.163 only speaks Responses API and llama.cpp translation triggers a 500 "System message must be at the beginning" from the Qwen3.5 template.
Takeaways: at this difficulty a 4B model suffices for basic agent work in most frameworks — the harness, not the model, is the bottleneck; error recovery matters more than raw capability. Author asks if anyone has tried sub-2B models.
More from coding & agent
- After testing 16 coding agents, an economist says escape the Codex/Claude Code bubble — felpix_ · 2026-10-21
- The joke command: Claude at max effort told to 'maximize monthly recurring revenue' — Kuprel · 2026-10-21
- Keeping Codex multi-agent teams lean: scope via pointers, not full-context prompts — umeshonai · 2026-10-21
- Ramp Router ships org-level controls as rivals pitch teams on dodging OpenRouter's 5.5-8% fees — KlausCodes · 2026-10-21
- Jev decision models add image inputs in preview, with new Box bounding-box question type — weswinder · 2026-10-21
- Infinitely forkable Zork: every reply is a save point in this multiplayer AI experiment — lfschiavo · 2026-10-21