LLM Quantization Explained: Bit Width Alone Doesn't Determine Accuracy or Speed

techNmak · x · 2026-10-19

A detailed thread (with infographic) by techNmak breaks down the engineering decisions behind LLM quantization, arguing bit width alone determines neither accuracy, memory, nor speed.

What gets quantized: weights, activations, and KV cache are separate precision choices (W4A16 = 4-bit weights + 16-bit activations).
Formats & granularity: INT4/INT8, FP4, FP8, and NF4 (nonuniform codebook) yield different errors at the same bit width; finer scale groups (per-channel/per-group) reduce error but add metadata and kernel constraints; outliers are the key error driver.
Algorithms: GPTQ uses approximate second-order info; AWQ protects salient weight channels via activation statistics; SmoothQuant redistributes difficulty to enable W8A8.
Training routes: PTQ (post-training + calibration), QAT (fake quantization), QLoRA (frozen 4-bit base + LoRA adapters).
Memory math: an 8B model costs 16 GB in FP16 vs 4.125 GB packed INT4 with scales; KV cache, activations, and buffers are extra.
Storage ≠ computation: W4A16 kernels typically dequantize tiles and run FP16/BF16 matmuls; real speedup depends on kernels, bandwidth, batch size, and workload.
Evaluation: measure held-out accuracy plus TTFT, per-token latency, throughput, and peak memory — a 4-bit model is not automatically 4x faster.

Original post →

LLM Quantization Explained: Bit Width Alone Doesn't Determine Accuracy or Speed

More from Infra