Swapping Qwen's N-gram layer to Q8 shows no real speed penalty in local inference tests
Altruistic_Heat_9531 · reddit · 2026-09-03
A local-LLM experimenter replaced the low-precision N-gram portion of an IQ4XS Qwen 3.8 Next model with Q8 weights (following someone else's BF16 experiment) and benchmarked on a Xeon E5-2690v4 + RTX 3090 (250W cap) with 96GB DDR4, no MTP.
Short-run generation jumped from 8.8 to 10.7 tok/s, but both setups converge to a steady state around 10.1 tok/s — meaning no meaningful speed penalty from the higher-precision N-gram layer. Output quality impact is still being tested.
More from Infra
- SGLang Summit sponsors grow from 11 to 36 in one week; Lip-Bu Tan, Lilian Weng to speak — BanghuaZ · 2026-10-18
- Yacine jokes his datacenter KPI is gigawatt-hours: get paged if not burning enough power — yacineMTB · 2026-10-18
- 1.68x inference speedup drops, directly accelerating GRPO post-training — yacineMTB · 2026-10-18
- OpenRouter is only a fraction of the total addressable market, poster argues — sloppenheimer · 2026-10-18
- Developer builds change alerts for LLM prices and retirements across Bedrock, Azure and Vertex — jsandhu0780 · 2026-10-18
- 12.3M tokens in 27 minutes for $0.14: agent session with 99.6% cache hit — teortaxesTex · 2026-10-18