Benchmarking 11 open models' post-GQA attention: up to 139x less memory at 128K
bariskasikci · x · 2026-11-21
Much of today's LLM inference analysis still assumes GQA, but recent open models have moved beyond it — combining latent caches, sliding windows, sparse attention, compressed caches and linear attention. The author used ServingStudio to compare 11 recent open models against the GQA baseline Qwen3-Coder-480B-A35B.
At 128K-token context, the new designs deliver:
- Memory: up to 139× less per request
- Decode: up to 250× fewer bytes read and 44× faster attention
- Prefill: up to 183× fewer attention FLOPs and 46× faster attention
These architectural differences drastically affect memory and compute efficiency — knowing what each model caches, reads, and where prefill/decode spend time is key to serving optimization.
More from Infra
- NVIDIA GTC DC lineup: Ian Buck, Catanzaro and Lebaredian sessions on infra, agents, physical AI — nvidia · 2026-11-21
- KV cache economics: why a $200 Claude plan can hand out $14,000 of API-equivalent tokens and still profit — rachittshah · 2026-11-21
- Baseten breaks down its security threat model as agent swarms accelerate attacks — baseten · 2026-11-21
- Nvidia's rivals are vying to be the 'Central Bank' of AI, says investor — GavinSBaker · 2026-11-21
- Anthropic files for Nasdaq IPO — why you should keep a local open-source LLM as backup — alex_verem · 2026-11-21
- Microsoft launches MAI Licensed Weights; Purdue researcher asks to run them on own hardware — marlene_zw · 2026-11-21