Bartowski Details New Per-Tensor Layout Maps for GGUF Quantization in llama.cpp

bartowski1182 · x · 2026-09-11

Quantization maintainer Bartowski published a Hugging Face blog on new per-tensor layout maps for GGUF quantization in llama.cpp.

Backstory: Upstream llama-quant.cpp handles MoE models with more than 8 experts poorly, and shexp tensors are surprisingly precision-sensitive despite being tiny; he has long maintained his own fork to cope.
The problem: Both upstream code and his tweaks are model-agnostic — shape-based heuristics like bumping bits in the first/last 1/8 of layers and every third middle layer only go so far.
New approach: Tensor-granular layout maps let bit allocation follow each model's actual structure. Research was largely powered by a Framework Desktop.

More from Infra