Qwen3.5-4B hits ~11.4 tok/s CPU-only on a Ryzen 5 laptop with llama.cpp tuning
solyarisoftware · x · 2026-10-29
- Developer TeksEdge built a standalone, thumb-drive-friendly .exe running Qwen3.5-4B (Q4KM GGUF) CPU-only on a Ryzen 5 7540U laptop (6C/12T, no GPU) at 11.4 tok/s.
- Key llama.cpp settings: 9 threads, --threads-batch 12, --prio 2, batch 2048/ubatch 512, flash-attn on, q40 KV cache, --repack, --mmap, all GPU offload disabled.
- Notable finding: MTP=3 was actually slower (10 tok/s); plain decode with 9 threads + Q4 KV won, yielding 10% speedup purely from tuning.
- Motivation: businesses are deploying local models to cut costs and preserve privacy.
More from Infra
- Beam integrates with Hugging Face to move million-scale datasets over its distributed network — markjeffrey · 2026-10-29
- Microsoft's 4,800-person layoffs save what its AI data center spend burns in two days — aakashgupta · 2026-10-29
- Strix Halo-specific kernels speed up vLLM prefill by up to 31x at long context — asfbrz96 · 2026-10-29
- FuriosaAI quadruples valuation to $2.9B in new funding round to challenge Nvidia — dinabass · 2026-10-29
- Dev questions Anthropic ops: Claude Code source leaked to prod, backbone outages 'routine' — cuenca · 2026-10-29
- System design for real-time AI: the 10 core components, from Kafka to edge inference — goyalshaliniuk · 2026-10-29