What breaks agents on a 4B model: 50 runs across 10 harnesses on Qwen3.5-4B

GapNew4766 · reddit · 2026-10-22

The author ran 10 agent harnesses (Pi, goose, Crush, Cline, OpenClaw, OpenCode, Hermes, Qwen Code, Codex CLI, Atomic Agent) on the same local Qwen3.5-4B (Q4KM, llama-server, 64K context, --jinja) on an M5 MacBook Pro 32GB, 5 small tasks each. Six harnesses passed all 5. Key findings:

Tool calls weren't the problem: grepping all logs found zero parse errors or unknown tools — failures came from elsewhere.
Exact-match edits are the real trap: renaming a function across 3 files failed because the oldtext skipped a blank line; the model resent the identical edit 5 times until the loop guard quit. Right idea, couldn't reproduce text byte-for-byte.
Small models overfit visible tests: a slugify fix passed visible cases but failed a hidden one (C++ & Rust 2026).
They miscount: counted 40 ERROR lines instead of 43 — counting belongs to grep -c.
Vague instructions followed literally: two agents wrote answers next to logs instead of the working folder.
11x speed spread on identical setup: 1m36s vs 17m43s; the fastest harness had 4 tools and a short system prompt.
Codex CLI never reached the model: Responses API only; llama-server's translation got rejected by Qwen's chat template with a 500.
No rogue behavior, but tasks were harmless, so it says little about safety.

Original post →

What breaks agents on a 4B model: 50 runs across 10 harnesses on Qwen3.5-4B

More from coding & agent