Dev benchmarks Gemma 4B on local agent tasks: scores 2/5, fails Extended mode threshold
draginol · x · 2026-11-08
draginol tested whether a 4B model (gemma-4-E4B-it-qat-q40 via Locama) can handle complex work using his newly released Local AI benchmark, which offers three tiers: Simple (80 classification items), Normal (turn a sales CSV into a working dashboard), and Extended (multi-step agent tasks in a virtual workspace, 15-20 min). The verdict screenshot shows the model scored 2/5 agent tasks at 61.5 tok/s decode — below the Extended threshold of 4-of-5 tasks and 15 tok/s. The benchmark also scores prefill, first token, total time, and peak memory.
More from Models
- MaxxBench sparks backlash: conflating benchmaxing with real spec adaptation — teortaxesTex · 2026-11-08
- JEPA debate: Berton says LeJEPA plateaus sooner than JEM, demands scaling laws — gabriberton · 2026-11-08
- Rumor: OpenAI shipped GPT-6 Astra only after completing a fresh pre-training run — haider1 · 2026-11-08
- Study tests continued pretraining on 1B-8B open models to refresh factual knowledge — burkov · 2026-11-08
- Prediction: GPT-6.2-Level Intelligence at 1,000 Tokens/Sec Next Year, Mass Humanoid Manufacturing — ChrisGPT · 2026-11-08
- Dev benchmarks tiny LLMs on agent tasks: 9B passes 2/5, stepping up to 12B — draginol · 2026-11-08