DeepSeek v4.1 Flash on-device test: q2 runs at 16 tok/s but tool calls go off the rails
challis88ocarina · reddit · 2026-09-12
- Running DeepSeek v4.1 Flash (q2) on an M3U 32/80c device: 300 tok/s prefill, 16 tok/s decode holding until 110k tokens — far better sustained speed than llama.cpp's steep drop-off, with GPU at 95% throughout.
- Tool calls work but choices are questionable: in a hermes workflow it ran find twice for a remote file locally, then ls /.ssh/ and grepped the entire remote machine despite the path already being in context; even 18k tokens of hermes prompt didn't rein it in.
- It ignores the code-execution tool (same as 4.0), making it less efficient than GLM 5.3 Flash (21-19 tok/s in comparison).
- Trade-off: big VRAM means waiting for big models; q4 gains look limited — bring on the M5U.
More from Infra
- effect-harness on Cloudflare: run agent loops, tools and credentials on separate VMs — samgoodwin89 · 2026-10-31
- Google's Project Suncatcher boots TPU in orbit and beams back first AI response — sundarpichai · 2026-10-31
- Together AI launches Post-Training platform for agentic RL with tool-use environments — togethercompute · 2026-10-31
- General Compute raises $100M for inference capacity as CoreWeave stock hits 12-month low — YvesMulkers · 2026-10-31
- Analyst warns of unstable AI compute feedback loop as Kioxia's NAND price warning triggers flash crash — bookwormengr · 2026-10-31
- SOMA launches enterprise prompt-compression layer claiming 20%+ inference savings — markjeffrey · 2026-10-31