llama.cpp PR adds IQ2_K-IQ6_K quants: same file size, much better KLD
ethertype · reddit · 2026-10-18
A new llama.cpp PR (#30715, +2101 lines by eaddario) proposes five new quantization types: IQ2K, IQ3K, IQ4K, IQ5K, and IQ6K.
- File sizes match the equivalent Q-quants — no extra disk or memory cost
- KLD improves substantially over Q-quants, meaning better fidelity at the same size
- Trade-offs: slightly worse runtime performance, and creating IQ quants is somewhat slow
- All numbers come from testing on a single model, so results may not generalize
Test models are uploaded to HuggingFace (eaddario/IQ2K-IQ6K-Test-Models) for local-deployment users to verify.
More from Infra
- Agent sandbox showdown: Solo.io compares NVIDIA OpenShell vs kagent + Agent Substrate on scale and HA — bibryam · 2026-10-18
- It's Time to Talk About TEEs: A Primer on Trusted Execution Environments — joshcsimmons · 2026-10-18
- Analyst argues market is badly wrong on Micron: HBM to grow 7x with prices tripling — JOBhakdi · 2026-10-18
- FT: 47 major US AI data center projects delayed or canceled amid $10T buildout — TansuYegen · 2026-10-18
- Google rewrites SRE book for the AI era, announces second edition 10 years later — mgill25 · 2026-10-18
- Agens Volundr 32B v1 Kills Repetition Loops; MIT Engine Fits 262K Context in 16GB — ComfortableKindly507 · 2026-10-18