Qwen3 4B scores 70/99 in a bare agent loop but 55/99 in LangChain Deep Agents with 4x tokens
SeveralSeat2176 · reddit · 2026-11-18
A developer ran a 99-task agent benchmark (tool chains, SQL, retrieval, files, multi-turn; checkers read tool-side state, not model replies) comparing harnesses on Qwen3 4B (Ollama, temp 0):
- A bare loop, createagent, and a hand-built LangGraph StateGraph all scored identically (70/99), while LangChain Deep Agents dropped to 55/99 while sending 3.6–4.3x more prompt tokens — planning and subagent prompts cost 15 tasks plus lots of context.
- Haiku 5.5 scored 99/99 under every harness, so scaffolding only hurts weak models.
- The same bench caught Qwen3 8B claiming it saved 150 when the store held 129: evals that only read final text will overrate small models.
- The whole run cost $0.93 across 2,589 calls, with every run recorded in a public write-up.
More from coding & agent
- Dev pairs Runway live avatars with Nous Hermes Agent to replace dot-and-blob agent UIs, plans to open source — alexcovo_eth · 2026-11-18
- Microsoft's Agent Framework PyPI Package Hacked to Steal Azure and GitHub Credentials — Kyrannio · 2026-11-18
- Grok-powered agent given $50 and a 'survive or shut down' challenge grew it to $5,763 in 48 hours — nikola_mr64990 · 2026-11-18
- Sable: an open-source desktop agentic dev environment wrapping Claude Code and Codex CLIs — churchkey · 2026-11-18
- OpenHands Teams launches: self-service agents to automate CVE remediation — rajistics · 2026-11-18
- Agent task trees deadlock under concurrency caps — slot lending fixes it — arpit_bhayani · 2026-11-18