Measured: lossy engine flags drop a Q8_0 model to roughly Q2_0 quality
jesdga95 · reddit · 2026-10-31
The author benchmarked two common lossy inference-engine flags (routing bias toward GPU-resident experts, tail skipping) on Qwen 3.8 Flash Next using their Basalt engine (RTX 5090 + 5060 Ti + 128GB DDR5):
- KLD: with shipped defaults, every quant pack lands between IQ3XXS and Q20; 178 GiB Q80 + shipped (KLD 0.401) degrades close to exact 65 GiB Q20 (0.436). Bigger packs suffer more since more experts sit outside VRAM.
- Real tasks: Q80 + shipped loses 1.5 points on closed-book facts (CI excludes zero); IQ3S + shipped on a small-card setup drops 5.2 on ladder tasks; worst-case config collapses facts from 38.1 to 2.7.
Takeaway: with ample VRAM the shipped defaults cost 1.5 points (4% of knowledge) for little speed gain; the tighter the VRAM, the worse the damage. Both flags are opt-in in Basalt for controlled measurement.
More from Infra
- Inference optimization race heats up: practitioners now want fast, cheap 'good enough' models, not the absolute best — bendee983 · 2026-10-31
- NVIDIA Releases Nemotron 3.5 Bolt E4B: 4GB On-Device Model with 80% Fewer Output Tokens — heatOverflower · 2026-10-31
- RTX 6000 Pro workstation GPU spotted at $13,999, down from ~$17K — HankYeomans · 2026-10-31
- Musk envisions lunar mass drivers launching compute to deep space for a petawatt-scale future — XFreeze · 2026-10-31
- DOE approves first commercial nuclear microreactor safety review; Radiant cleared for full-power testing — McDonaghMatthew · 2026-10-31
- Dev shows off eye-popping LLM cache hit rate: inference costs plummet — sujingshen · 2026-10-31