Local LLM quantization guide: Hardware thresholds for FP8, NVFP4, and more
Ill_Dragonfruit_3547 · reddit · 2026-08-20
The author shares lessons on quantization and hardware compatibility for running local LLMs, emphasizing that quant formats are tied to specific hardware generations, not just compression ratios.
Hardware Generations & Formats:
- Turing (RTX 20): FP16, INT8, INT4
- Ampere (RTX 30, A100): +BF16, TF32 (No native FP8)
- Ada Lovelace (RTX 40): +FP8 (E4M3/E5M2)
- Hopper (H100): +FP8, Transformer Engine
- Blackwell (RTX 50): +FP4 (NVFP4), FP6
Format Taxonomy:
- Portable (Safe): GGUF (universal), MLX (Apple Silicon), fp16/bf16 safetensors (universal).
- Hardware-Specific: FP8 (Ada/Hopper, slow on Ampere), NVFP4 (Blackwell only), EXL2/EXL3 (CUDA only), AWQ/GPTQ (CUDA only).
Key Rule: On an M1Max, stick to MLX for LLMs and GGUF/fp16 safetensors for diffusion. Other formats are likely to fail or perform poorly.
More from Infra
- ML Researcher: Compute Shortage, Not Ideas, Slowing Down Progress — A_K_Nain · 2026-08-20
- M3 Ultra achieves 45.8% performance gain running 27B model with CPU+GPU+ANE offload — AIFlow_ML · 2026-08-20
- Llama.cpp restarts kill long context prefills: proposal for auto KV state persistence — Dazzling_Equipment_9 · 2026-08-20
- YC-backed Atomarine builds floating data centers to solve the compute crunch — ycombinator · 2026-08-20
- Animation puts AI data center water usage in context amid debate — ATTlKA · 2026-08-20
- Mojo integrated with MLIR stack, running matrix multiplication on Corsair in days — clattner_llvm · 2026-08-20