ISTA-DASLab's NVFP4 prefiller recovers Qwen3.8-27B low-quant accuracy with SSD streaming
magnus-m · reddit · 2026-10-20
ISTA-DASLab released a prefiller approach for Qwen3.8-27B low-quant weights: a NVFP4 prefiller model streamed from SSD recovers accuracy at aggressive quantization levels (IQ1S to IQ3XXS) while keeping low VRAM usage and fast decoding. Benchmark charts show large gains, biggest at low bits (e.g. IQ1S from 30% to 62%).
Caveats: requires an Nvidia Blackwell GPU; no speed penalty vs no-prefiller for contexts beyond 4k. Weights are on Hugging Face (ISTA-DASLab/Qwen3.8-27B-NVFP4-prefiller). The poster has not tested it himself.
More from Infra
- doubleAI, co-founded by Mobileye CEO, claims its model cuts compute costs by 67% — pstAsiatech · 2026-10-20
- US sanctions 5 Chinese entities tied to ex-Google engineer who stole TPU cluster secrets — rohanpaul_ai · 2026-10-20
- AI chip startup Tenstorrent raises $1.5B at $6.2B valuation after takeover interest — pstAsiatech · 2026-10-20
- Fresher Lemonade: CLI tips for pulling, configuring and benchmarking local models — admcpr · 2026-10-20
- BigMoeOnEdge streams 91B+ MoE models on 12GB phones and 16GB Macs, hitting 25-30 tok/s — dai_app · 2026-10-20
- NVIDIA Brev lets you add your old gaming PC to a unified GPU console alongside cloud instances — victorialslocum · 2026-10-20