One RX 7900 XTX runs 524k context at 767 tok/s, 3x faster than llama.cpp

W61k3r · reddit · 2026-11-22

A developer ran Inkling-2-Mini entirely on a single 24 GB RX 7900 XTX using their own Rust inference engine (LexiSuite), achieving a 524,288-token context with 23.7/25.75 GB VRAM in use and a 13-second load time.

Key numbers (vs llama.cpp b12600, same card and file):

Needle retrieval: 10/10 facts recovered at both 260k and 523k tokens.

The draft head is a big win at short/medium depth but slightly slower at 500k+ depth, as its own attention costs more than it saves. Accuracy matches llama.cpp (KL 0.004 vs CPU reference, error mainly from the 4-bit KV cache). The 45% prefill gain since last version came from a new attention kernel for the global-attention layers. 1M context doesn't fit on this card.

Related event: Custom Rust engine runs 524k context on a single RX 7900 XTX(2 posts)→

Original post →

More from Infra

Infra channel →