Local model agentic coding benchmark: 27B matches 125B, tool-calling is the lever
Joralk_ · reddit · 2026-10-23
The ggufbench author benchmarked local GGUF models for agentic coding with one harness: 18 tool-calling, 72 code and 72 agent tasks, 3 seeds each, pinned sampling, 2×RTX 5080.
- Surprise: MiniCPM-V-4.7-35B (0.85) and Qwen3.8-27B (0.82) tie with the 125B Qwen3.8-Flash-Next on the Strata engine (0.80); the 125B wins nothing. Size isn't the lever for agentic coding — tool-calling reliability is, and the 27B class already saturates it.
- Thinking mode is a tax: on the 27B it lifts code 0.79→0.93 and agent 0.90→0.96, but latency goes 36ms→629ms with +253 median tokens per call; keep it off by default, on for hard tasks.
- Engine made no measurable difference (0.920 vs 0.923 with BigMoeOnEdge), though it surfaced a router-hook bug returning empty messages, filed with a backtrace.
- Limitations: consumer GPUs, self-built task set, one machine, one quant per model. Live scoreboard and open-source repo available.
Related event: Local GGUF benchmark: 35B models rival 125B in agentic coding(2 posts)→
More from coding & agent
- Most new CVEs hit hand-written legacy code, dev pushes back on vibe coding security blame — HowDevelop · 2026-10-23
- Claude and Claude Code Merge: Anthropic Engineers Say Delete Your CLAUDE.md Every 6 Months — rubenhassid · 2026-10-23
- Engineers merging 4x more PRs at Imprint: Will Larson on speeding up code review — AxSaucedo · 2026-10-23
- 15 practical ways to hand off work to Grok Bot's cloud computer — even with laptop closed — XFreeze · 2026-10-23
- 3 Lessons from Rebuilding Compute for AI Agents: Safe Sandboxes, Near-Free Idle, Persistent Memory — Saboo_Shubham_ · 2026-10-23
- Meta's RSD paper: agents self-improve from unlabeled runs, lifting Qwen3-14B success from 12.9% to 20.6% — rohanpaul_ai · 2026-10-23