425MB Qwen3.5-0.8B decision model on CPU: custom 4-bit quant loses just 0.18pp, 2x faster

Danmoreng · reddit · 2026-11-22

Developer Danmoreng open-sourced qwen35-cpu, an optimized CPU engine for a finetuned Qwen3.5 0.8B decision model. A calibrated custom 4-bit quant (424.9MB) scores 68.74% macro accuracy vs 68.92% for BF16 across 13,601 scored questions, while running over 2x faster: 50ms latency on a Ryzen 9 9955HX3D with short inputs (vs 81.1ms BF16) and 222.9ms at 579 tokens (vs 546.5ms), 218ms on an older Intel 8750H. Calibration adds 1.1pp accuracy at negligible latency cost. A complete, reproducible reference for on-device small-model deployment.

Original post →

More from Infra

Infra channel →