Dev benchmarks tiny LLMs on agent tasks: 9B passes 2/5, stepping up to 12B
draginol · x · 2026-11-08
A developer is running a hands-on series testing whether small-parameter models can handle complex agent tasks: after finding the 4B model inadequate, he reports the 9B version (Ornith) still isn't powerful enough for extended work and is moving up to 12B.
The benchmark screenshot shows the 9B model passing 2 of 5 tasks (average score 0.73 across 40 steps):
- Passed: inventory backorder workflow (0.94), SaaS churn recovery (0.82, hard difficulty)
- Failed: budget risk summary (0.90 but not accepted), margin decline driver analysis (0.69), and a JS zero-value serialization coding task (0.27, exceeded the hard action limit of 11 steps)
The takeaway: small models remain unreliable on tool-use and coding agent tasks, frequently blowing past step limits, and likely need 12B+ scale to hold up on extended work.
More from coding & agent
- Open-source satmux boosts inflight GEO Wi-Fi up to 6.4x with $0.03/h VM relay — steipete · 2026-11-08
- IV Studio 1.0: Open-Source ComfyUI Film Studio Adds Linux, Mac and Local LoRA Training — itsVariance · 2026-11-08
- Dev builds research wiki for agent swarms, following Anthropic's wiki memory notes — teortaxesTex · 2026-11-08
- Nadella: Business apps will collapse in the agent era, becoming dumb databases — _AustinCalvert_ · 2026-11-08
- Reddit dev asks how to build an LLM-agnostic agent stack and escape Claude lock-in — DataBeek · 2026-11-08
- RL env startups urged to focus on finance; Anthropic ships Claude for Financial Services — himanshustwts · 2026-11-08