Custom CPU engine for Qwen3.5 0.8B beats llama.cpp with 2.9x faster prefill
Danmoreng · reddit · 2026-09-08
Reddit user Danmoreng vibecoded a small C++ inference engine and a custom 4-bit quant format (H128/Q4-G32-DOT4, with activation-based calibration and blockwise error compensation) to run Qwen3.5 0.8B on CPU — his use case is local dictation cleanup while the GPU is busy.
- 425 MB weight payload, 71 MB smaller than Unsloth's mixed-precision Q40, with similar perplexity and KL divergence
- On a Ryzen 9 9955HX3D (8 V-Cache cores) vs llama.cpp Q40: 2.9x prefill, 1.3x single-request decode, 1.7x batch-16 throughput
- vs ikllama.cpp on the same GGUF: 18% faster single-request decode, 53% higher batch-16 throughput
- Open source at github.com/Danmoreng/qwen35-cpu
More from Infra
- Polymarket puts 74% odds on a US state enacting a data center moratorium by end of 2026 — Polymarket · 2026-09-08
- Denver data center filmed heavily watering lawn while 1.5M residents face drought restrictions — Polymarket · 2026-09-08
- One mental model for Kubernetes, Slurm, Ray, and Spark: a unified take on distributed compute — ArchitectingAI · 2026-09-08
- Aurora Fork Fixes OpenCode API HTTP 400 Errors and Cuts Token Costs Up to 80% — entitybtw · 2026-09-08
- NVIDIA Launches Free Tool to Turn Your PC Into a Personal AI Data Center — ohiocodernumerouno · 2026-09-08
- ExLlamaV3 Is Underrated: Better Quants, Lower KLD, Faster Than llama.cpp — Embarrassed_Soup_279 · 2026-09-08