ISTA-DASLab splits prefill/decode quantization, squeezes 27B-class model into 13.7GB GGUF
victormustar · x · 2026-09-21
ISTA-DASLab released an experimental HF repo, Qwen3.8-27B-disaggregated-NVFP4-prefill, premised on the idea that prefill and decode have different bottlenecks and needn't share one quantization format:
- Prefill uses NVFP4, well suited to fast low-precision matrix ops.
- Decode uses a more compact representation (IQ1–IQ2) to cut memory bandwidth needs.
- The prefill representation appears to be streamed/loaded on demand instead of keeping two full copies in GPU memory.
- Total GGUF size is about 13.7GB for this 24B-param model.
It remains experimental: no model card, custom odp architecture, and not a drop-in GGUF for existing runtimes.
More from Infra
- NVIDIA's next gaming GPU may slip to 2028 as Rubin capacity goes to AI — mark_k · 2026-09-21
- AMD GPU + ComfyUI: local Minimax H3 video gen upscaled to 1080p on RX 9070 — mwhjose · 2026-09-21
- Usage limit killed scheduled agent jobs for 19 hours — team shares 4 fixes so agents monitor themselves — lilythemoon54 · 2026-09-21
- Devs say avoiding cache misses could boost Claude Code/Codex effective usage limits 10-20% — chaseleantj · 2026-09-21
- How do you test LLM provider failure in production? A Reddit discussion — Rama_Surasani_ · 2026-09-21
- Run Flux 2 Dev (30B) locally with block offloading, mix models for T2I and editing — Altruistic_Heat_9531 · 2026-09-21