4-bit keeps 98-99% quality, DeepSeek V4 Flash isn't a local-deployment model
gnukeith · x · 2026-10-14
Developer gnukeith shared quantization rules of thumb: 4-bit retains 98-99% of original quality, 8-bit is near lossless, while 3-bit and below often drops into the low-to-mid 90s or worse depending on model and task — he warns running models at 2-3 bit will hurt you beyond just cost. He cites MiMo-V2.6's use of MXFP4 quantization-aware training (QAT) during mid-training to preserve quality at low precision. He also argues DeepSeek V4 Flash isn't realistically runnable at home despite aggressive KV cache compression, pointing to Qwen 3.8 Flash-Next or MiMo 2.6 Flash as the likely local models.
More from Infra
- Xinjiang's 167 GW green power, 16ms to Chongqing: the grid edge in AI race — 96Stats · 2026-10-14
- Migrating a key API from Node to Bun cut p99 latency 80% and memory 63% — DanielLockyer · 2026-10-14
- Agent traffic is a strange inference workload: long prefix caching beats raw tokens per second — rohanpaul_ai · 2026-10-14
- LightVAE speeds up video VAE decoding up to 2.24x on H100 with minimal quality loss — Shiqiao Gu · 2026-10-14
- Orbital data centers have no off-switch: Reddit debates AI safety paradox — Zappy_Oh · 2026-10-14
- Rednote's Dots unveils MSAR: 1.58x iso-loss compute speedup over residual baseline — teortaxesTex · 2026-10-14