Visualized: GLM 5.3 decode offload reads 13GB of weights plus experts per token
HankYeomans · x · 2026-11-11
@nettermina's "Offload visualized — decode edition" (final entry of the series) breaks down VRAM offloading during GLM 5.3 decoding:
- Writing a single token requires reading 13GB of always-resident weights plus 8/256 experts for all 75 layers.
- In the example, copies of 7/8 expert picks are kept on GPU (expert cache hits); misses are computed on CPU directly from RAM, with some copied back to GPU where most efficient.
A clear visualization of MoE expert-cache hit rates and CPU/GPU division of labor for local inference.
More from Infra
- KubeRay joins CNCF as the standard way to run Ray workloads on Kubernetes — PyTorch · 2026-11-11
- Hugging Face launches Local AI 301 course on quantization — huggingface · 2026-11-11
- PyTorch megakernel runs 816B MoE on one TPU v7 host at 433 tok/s, beats GB200 on cost — PyTorch · 2026-11-11
- Can local AI kill the Akamai thesis? NPUs ship in a third of new devices, but hybrid wins for now — pdamodaran · 2026-11-11
- AWS official guide: build a Bedrock invocation observability data lake in five stages — AWS ML Blog · 2026-11-11
- EU servers don't guarantee AI sovereignty — TrueFoundry launches self-hostable AI Gateway in EMEA — azed_ai · 2026-11-11