Forked llama.cpp runs 64K-context model at 60+ tok/s on one 12GB GPU, 17x faster prefill
Puzzled-Stick9170 · reddit · 2026-11-22
A Reddit user details getting Inkling 2 Mini running at 51-64 tok/s decode and 1,500 tok/s prefill with 64K context on a single RTX 4070 Ti (12 GB), versus 29-34 tok/s with upstream llama.cpp and CPU expert offloading.
Key optimizations in the inkling-fast fork:
- Loop-aware 2-3 bit quant (mixB): expert down-projection rows of 1152 aren't multiples of 256, silently falling back to IQ4NL upstream; adding --pad-exps zero-pads rows to 1280, and extra bits go to the loop core (layers 6-17) that runs 3x per token. File size drops to 10.3 GiB.
- Fusing relative-position bias into the CUDA Flash Attention kernel: cuts the 64K-context compute buffer from 6.5 GB to 1 GB, driving prefill from 88 to 1,500 tok/s.
- Fewer kernel launches (5,000 → 3,500): decode on a 12 GB card turns out to be kernel-launch bound, not bandwidth bound — all-IQ2XXS and all-IQ2S decode identically. New fused ops plus static-graph caching lifted the same file from 54 to 63.5+ tok/s decode.
Quality holds at 90.9% top-1 match and 0.048 KLD vs Q80. Chat, tool calling, vision (mmproj on CPU) and prompt caching all work. Pitfalls documented: no gain from DSpark speculative decoding, f16 KV doesn't fit at 64K (q80 required), and 5 tok/s lost per expert layer offloaded to CPU.
More from Infra
- Google AI members get free monthly Cloud credits: $10/mo for Pro, up to $100/mo for Ultra 20x — lxfater · 2026-11-22
- Inkling 2 mini's 1M-context KV cache dissected: only 8 of 48 layer visits keep full cache — teortaxesTex · 2026-11-22
- Musk says compute is going to Optimus; cars will remain at AI4 — CyberRobooo · 2026-11-22
- Comet cut Opik dataset ingestion time from 39 minutes to 80 seconds by fixing 8 bottlenecks — dl_weekly · 2026-11-22
- Ghost Mode: open-source Rust Code Mode cuts ~60% of agent tool-call token costs — legend-of-alan · 2026-11-22
- Checkpoint & restore for vLLM on AMD Instinct GPUs detailed in new engineering post — kbesplv · 2026-11-22