Custom Rust engine runs 524k context on one RX 7900 XTX, 5x faster than llama.cpp

W61k3r · reddit · 2026-11-22

Developer W61k3r benchmarked his Rust inference engine LexiSuite running Inkling-2-Mini on a single 24GB RX 7900 XTX: 2,245 tok/s prompt and 52.4 tok/s generation at 4k context vs llama.cpp's 425/25.4 on the same model file. At 262k context LexiEngine hits 1,146 tok/s prompt while llama.cpp failed to finish in 55 minutes. At 524k tokens: 767 tok/s prompt, 33.7 tok/s generation, with 17.6GB weights + 5GB 4-bit KV cache fitting in 23.7/25.75GB VRAM, 13s load time. Needle retrieval is 10/10 at both 260k and 523k depths; accuracy matches the CPU reference (mean KL 0.0038), with residual error from the 4-bit KV cache, not bugs. The built-in draft head speeds up short/medium contexts (79 vs 43 tok/s at 16k) but slightly slows things at 500k+ depth. 1M context doesn't fit on 24GB.

Related event: Self-Built Rust Engine Runs 524k Context on a Single GPU, Several Times Faster Than llama.cpp(3 posts)→

Original post →

More from Infra

Infra channel →