Forked llama.cpp runs 64K-context model at 60+ tok/s on one 12GB GPU, 17x faster prefill

Puzzled-Stick9170 · reddit · 2026-11-22

A Reddit user details getting Inkling 2 Mini running at 51-64 tok/s decode and 1,500 tok/s prefill with 64K context on a single RTX 4070 Ti (12 GB), versus 29-34 tok/s with upstream llama.cpp and CPU expert offloading.

Key optimizations in the inkling-fast fork:

Quality holds at 90.9% top-1 match and 0.048 KLD vs Q80. Chat, tool calling, vision (mmproj on CPU) and prompt caching all work. Pitfalls documented: no gain from DSpark speculative decoding, f16 KV doesn't fit at 64K (q80 required), and 5 tok/s lost per expert layer offloaded to CPU.

Original post →

More from Infra

Infra channel →