425MB Qwen3.5-0.8B decision model on CPU: custom 4-bit quant loses just 0.18pp, 2x faster
Danmoreng · reddit · 2026-11-22
Developer Danmoreng open-sourced qwen35-cpu, an optimized CPU engine for a finetuned Qwen3.5 0.8B decision model. A calibrated custom 4-bit quant (424.9MB) scores 68.74% macro accuracy vs 68.92% for BF16 across 13,601 scored questions, while running over 2x faster: 50ms latency on a Ryzen 9 9955HX3D with short inputs (vs 81.1ms BF16) and 222.9ms at 579 tokens (vs 546.5ms), 218ms on an older Intel 8750H. Calibration adds 1.1pp accuracy at negligible latency cost. A complete, reproducible reference for on-device small-model deployment.
More from Infra
- A Hands-On GPU Profiling Starter Path: Time PyTorch Attention with Stanford CS336 Assignment 2 — stanfordnlp · 2026-11-22
- 42 Sellers, 5x Price Spread: Open Models Enrich Big Inference Providers, Not Startups — AccBalanced · 2026-11-22
- Inkling 2 Mini tops efficiency charts: 26B looped transformer at $25.80 per correct answer — TheZachMueller · 2026-11-22
- Gemini 4 Neon autonomously writes GPU kernels, 5.9x faster prefill on AMD hardware — sebkrier · 2026-11-22
- SenseNova-Loop-9B Hits 1,300+ tok/s on a Laptop RTX 5090, Benchmarked vs Qwen 27B — tebjan · 2026-11-22
- Intel's 350W D270 claims 5.9x lead over RTX PRO 6000 — but the fine print reveals benchmark games — StefanoGogioso · 2026-11-22