Sub-4B model agent benchmark: Qwen3.5-4B hits 51.4% strict pass, thinking mode adds up to 20 points
KienShen · reddit · 2026-11-04
CamThink evaluated six sub-4B models on agent tasks using their NeoMind benchmark (138 bilingual zh/en tasks; strict pass = required command executed AND post-run state assertions passed):
- Strict pass rates: DeepSeek-flash 81.2% and GLM-5.3 76.8% (cloud API) lead; local models: Qwen3.5-4B·T1.0 51.4%, Spark-X2.6-4B 50.0%, LFM3-1.7B 42.8%, LFM2.5-2.6B 36.2%, Gemma-4-E2B just 26.1%.
- Thinking mode: consistently large gains—Spark-X2.6-1.7B jumps from 43.5% to 63.8%, Qwen3.5-4B from 41.3% to 55.1%.
- Multi-step tasks: all models drop sharply on multi-step work; LFM2.5-2.6B manages only 12.5% multi-step vs 45.9% single-step.
Takeaway: small-model intelligence density is improving fast, but multi-step agency remains the weak spot. Setup: NeoMind 1.2.3, 32K context, Q4 quantization.
More from coding & agent
- The agent-tool gold rush: sell traces to labs once, sell failed trajectories as RL envs twice — himanshustwts · 2026-11-04
- DataFlow PRs add bad-line isolation and sample-level retries for dataset cleaning — Puzzleheaded_Box2842 · 2026-11-04
- Telemetry spat continues: dev presses Theo on hiding the opt-out in an env var — uwukko · 2026-11-04
- 590 Claude Code sessions, 20B tokens later: a 30-hour browser RPG built almost entirely by agents — Comprehensive-Dig-31 · 2026-11-04
- Theo responds to T3 Code telemetry flap: fully disableable via env var — theo · 2026-11-04
- shadcn hails Claude Opus 5.5 as the first model with 'taste' — shadcn · 2026-11-04