Does FP8 KV Cache degrade Qwen3.8-27B quality in long contexts?
Valuable-Run2129 · reddit · 2026-08-20
The author discusses the trade-off of using FP8 KV Cache on Qwen3.8-27B to enable MTP (Multi-Token Prediction) capabilities in long-context scenarios.
Core Dilemma:
- Common advice is to avoid quantizing the cache, but with 260k context lengths, FP8 is necessary to fit MTP.
- The author seeks verification on whether FP8 quality degradation is negligible compared to FP16, as AI suggests it might be.
Focus:
- Whether FP8 quantization causes noticeable capability degradation in extreme long-context generation.
- Finding the balance between inference speed (enabling MTP) and model quality.
More from Infra
- Large data centers employ only ~50-150 permanent on-site staff, Microsoft says — brezina · 2026-10-16
- Prediction: Opus 5.5-level models on laptops by 2027, backed by Qwen 27B data — cephaloform · 2026-10-16
- Oracle down 12% in a week, nearly 50% off June high as OpenAI exposure spooks market — alvelda · 2026-10-16
- US data center moratoriums top 400 — an AI bull explains why they're actually good for AI — kevinsxu · 2026-10-16
- Microsoft Research unveils Magnetar, the fastest zkVM at 10-39x speedups and post-quantum security — jedisct1 · 2026-10-16
- Bonsai 2 2B hits 120 tok/s on iPhone 18 Pro, outpaces Qwen 3.5 2B at half the memory — cephaloform · 2026-10-16