96-Head MLA Hits 363 FLOPs/Byte Compute Intensity; FP8 KV Still Can't Escape Compute Bound
YouJiacheng · x · 2026-10-26
YouJiacheng and others discussed the compute/bandwidth trade-offs of MoE and linear projection: whether core attention is compute bound is independent of batch size, and the point of FP8 KV is to avoid OOM at large batch sizes—during computation it still gets dequantized to BF16 (see FlashMLA).
More from Infra
- RTX 5090 hits $7K as PC gaming community goes into 'denial' over AI boom — PAstynome · 2026-10-27
- Wan2GP lands on Pinokio: 1-click AI video on 6GB GPUs — cocktailpeanut · 2026-10-27
- AWS on agent prompt caching: explicit + implicit patterns to slash token costs — AWS ML Blog · 2026-10-27
- Thieves spoofing GPS steal $100M+ in data center cargo since June — Ars Technica AI · 2026-10-27
- Lam Research: weekly token usage up ~250x in 18 months — Beth_Kindig · 2026-10-27
- AWS shows how to build enterprise agentic retrieval on Bedrock with CDK — AWS ML Blog · 2026-10-27