Local 4B model fails Extended agent benchmark: 2/5 tasks at 61.5 tok/s
draginol · x · 2026-11-08
Continuing his local AI benchmark testing, draginol had a 4B model (gemma-4-E4B-it-qat-q40) attempt Extended-mode multi-step agent tasks — and it failed. The verdict shows 2/5 tasks passed at 61.5 tok/s decode, short of the 4-of-5 tasks and 15 tok/s threshold. He says he'll move on to an even smaller model next.
Related event: Local Gemma 4B Model Fails High-Difficulty Agent Benchmark(2 posts)→
More from Models
- Vercel lists DeepSeek V4.1 Flash 1017 at 1,000+ tok/s, but provider speed numbers don't add up — gajesh · 2026-11-08
- Researchers spar over why JEPA is hyped as 'the big thing in vision' among non-experts — ducha_aiki · 2026-11-08
- Kimi K3.1 Pro tops BabyVision image benchmark at 95.4%, beating Inkling 2 Small — NielsRogge · 2026-11-08
- World model licensing is a mess: open weights, restrictive decoders block commercial use — prateekj · 2026-11-08
- Gemini Won't Accept PPTX Files, but NotebookLM Does — Users puzzled — Pearl_Jam_ · 2026-11-08
- giffmana: paper claims zero real savings—no scaling trend or FLOPs-matched results — giffmana · 2026-11-08