CPU-Only LLM Tests: 35B MoE at Q2 Beats a 2B Model Despite Half the Speed
ML-Future · reddit · 2026-09-08
The author debates the future of local LLMs — tiny-but-smart vs. huge-but-optimized — with hands-on CPU-only benchmarks.
- MiniCPM5 2B Q8: 6 t/s without vision, but makes many mistakes
- Qwen3.6 35B (MoE) Q2XXS: 3 t/s but far more impressive, usable results
- Verdict leans toward heavily-quantized MoE large models over small dense ones for CPU-only setups
Follows up the author's earlier post on running Qwen3.6 35B without a GPU.
More from Infra
- Cloud capacity crisis is now an AI architecture problem, argues new newsletter — DavidLinthicum · 2026-10-26
- SpaceX files $40B GPU financing commitment with Apollo, already at 1.7GW — XFreeze · 2026-10-26
- Meta plans to raise up to $95B to fund massive AI expansion — Polymarket · 2026-10-26
- Nebius CEO: Debt Markets Closed for Neoclouds Last Week, Leaning on Customer Advances — zephyr_z9 · 2026-10-26
- CoreWeave 5-year CDS spikes to ~1,100bps as market fears AI bubble starts as credit event — kimmonismus · 2026-10-26
- AI demand reshapes cloud computing: scarce capacity and long-term deals replace on-demand — DavidLinthicum · 2026-10-26