Bartowski Details New Per-Tensor Layout Maps for GGUF Quantization in llama.cpp
bartowski1182 · x · 2026-09-11
Quantization maintainer Bartowski published a Hugging Face blog on new per-tensor layout maps for GGUF quantization in llama.cpp.
- Backstory: Upstream llama-quant.cpp handles MoE models with more than 8 experts poorly, and shexp tensors are surprisingly precision-sensitive despite being tiny; he has long maintained his own fork to cope.
- The problem: Both upstream code and his tweaks are model-agnostic — shape-based heuristics like bumping bits in the first/last 1/8 of layers and every third middle layer only go so far.
- New approach: Tensor-granular layout maps let bit allocation follow each model's actual structure. Research was largely powered by a Framework Desktop.
More from Infra
- Grid queues make power architecture an AI-factory decision; Exowatt and Redwood offer fixes — ingliguori · 2026-10-27
- Two CVSS 9.9 Wasmtime flaws allow sandbox escape, one ARM64-only since 2020 — basedjensen · 2026-10-27
- AI Infra Meme: While Countries Need TW-Scale Power for Datacenters, People Argue About Lightbulbs — mgostIH · 2026-10-27
- NVIDIA reportedly stops taking RTX 5090 orders from board partners, even backorders — chemist_slime · 2026-10-27
- Legora runs 2,000 parallel agent evals on Modal Sandboxes for legal AI — blelbach · 2026-10-27
- Cloud release velocity scores: AWS 95, Azure 90, Google Cloud 82 — DavidLinthicum · 2026-10-27