Voodoo Dynamic Quant Goes MIT: Gradient Descent Picks Per-Tensor Quant Levels
1ncehost · reddit · 2026-09-15
A Reddit author open-sourced Voodoo Quant under MIT after it hit SOTA at aggressive quant levels on small Qwen3.5 GGUF models. The method is the first to use gradient descent to optimize per-tensor quant layouts: candidate quant levels run simultaneously with frozen weights while a scalar gate per tensor per level is trained, annealed via tau and softmax, with a loss combining KL divergence to the BF16 reference and filesize-target reward. In testing, Unsloth Dynamic 3.0 wins at mid-to-high quants, but VQ wins at the most aggressive levels. Research-grade, untested at larger sizes; full toolset on GitHub, currently set up for Qwen.
More from Infra
- ESP32-powered 4-inch touchscreen builds a TUI dashboard for monitoring a DGX inference cluster — CATLLM · 2026-11-02
- OpenRouter crosses 10 quadrillion annualized token run rate, doubling every 10.5 weeks — deedydas · 2026-11-02
- Palmer Luckey asks: can you use datacenter RAM for things prohibited by the operator's terms of service? — avaitopiper · 2026-11-02
- Disable MTP for 20-30% faster local LLM inference on limited RAM — InternalMode8159 · 2026-11-02
- Kioxia up 216% in a year but down 68% from its June 2026 peak — zephyr_z9 · 2026-11-02
- Intel plunges 29%, worst day since the 1970s, wiping $12B off US government stake — emmanuelvivier · 2026-11-02