Inkling 2 mini's 1M-context KV cache dissected: only 8 of 48 layer visits keep full cache
teortaxesTex · x · 2026-11-22
A breakdown of how the new Inkling 2 mini maintains a comparatively small KV cache at 1M-token context:
- Only 8 of its 48 layer visits keep a full KV cache: 20 of 24 layers use a 1024-token sliding window, and the 4 full-attention layers are indices 5, 11, 17 and 23.
- Layers 11 and 17 sit inside the loop and run 3 times each, totaling 8 full-cache visits.
- At fp8 that's 16 KiB per token, 16 GiB at 1M context — versus 96 GiB if all 48 visits kept a full cache.
The author had hoped to serve 4 sessions off one RTX 4090, but notes a certain 552B MoE reportedly performs 18x better.
More from Infra
- Musk says compute is going to Optimus; cars will remain at AI4 — CyberRobooo · 2026-11-22
- Comet cut Opik dataset ingestion time from 39 minutes to 80 seconds by fixing 8 bottlenecks — dl_weekly · 2026-11-22
- Ghost Mode: open-source Rust Code Mode cuts ~60% of agent tool-call token costs — legend-of-alan · 2026-11-22
- Checkpoint & restore for vLLM on AMD Instinct GPUs detailed in new engineering post — kbesplv · 2026-11-22
- HN debate: Is the AI boom becoming a disaster for the climate? — Brajeshwar · 2026-11-22
- Moore Threads rumored 64GB consumer GPU "Lushan" would double RTX 5090 VRAM — CeFurkan · 2026-11-22