Inkling 2 mini's 1M-context KV cache dissected: only 8 of 48 layer visits keep full cache

teortaxesTex · x · 2026-11-22

A breakdown of how the new Inkling 2 mini maintains a comparatively small KV cache at 1M-token context:

The author had hoped to serve 4 sessions off one RTX 4090, but notes a certain 552B MoE reportedly performs 18x better.

Original post →

More from Infra

Infra channel →