Benchmarking 11 open models' post-GQA attention: up to 139x less memory at 128K

bariskasikci · x · 2026-11-21

Much of today's LLM inference analysis still assumes GQA, but recent open models have moved beyond it — combining latent caches, sliding windows, sparse attention, compressed caches and linear attention. The author used ServingStudio to compare 11 recent open models against the GQA baseline Qwen3-Coder-480B-A35B.

At 128K-token context, the new designs deliver:

These architectural differences drastically affect memory and compute efficiency — knowing what each model caches, reads, and where prefill/decode spend time is key to serving optimization.

Original post →

More from Infra

Infra channel →