LookAtMyQuant: a quantization compiler with 12x lower KL at Q4_K_M size on Gemma 4 E2B
teortaxesTex · x · 2026-11-02
Developer shinboson released LookAtMyQuant, a quantization compiler for language models. Insight: most existing quantization methods are grab bags of the same operators; the author found a shared grammar of composable tensor-network rewrites, measures how much compressing each piece (weight matrices, embedding buckets by token frequency, per-layer KV cache) hurts the model, and assigns each piece the code that minimizes total damage under a device size/speed budget.
Key results:
- Nine published quantization/pruning methods reproduce exactly as short programs in this grammar
- On Gemma 4 E2B, restricted to llama.cpp types it makes ordinary GGUF files: at Q4KM size, 12x lower KL divergence, 59% on GSM8K vs 50% for Q4KM (61% original)
- Most gain comes from Gemma's 2.35B-weight per-layer lookup table: cheap to compress, stored at 4.25 bits instead of 6.56, with saved bytes given to matrices
- On Qwen3.5-4B and SmolLM2-360M, plans show 6–81x lower KL at equal cost vs 3-bit baselines
teortaxesTex highlights the compiler concept and per-layer lookup table gains, wondering about inference speed.
More from Infra
- AI Defense Agents Running 24/7 Could Send Token Costs Through the Roof — robleclerc · 2026-11-02
- AI Data Centers Near a PC-Style Inflection Point: OCP Analysis of Power, Cooling, Networking — ai · 2026-11-02
- Cerebras halves in 5 months post-IPO: $25.4B order book leans heavily on OpenAI — FinanceYF5 · 2026-11-02
- Together launches post-training platform aligning RL training with serving — togethercompute · 2026-11-02
- Demis Hassabis: AI data centers should be beautiful — Recent_Fox4339 · 2026-11-02
- Ava: an open-source real-time voice avatar that runs fully on-device in your browser — victormustar · 2026-11-02