Custom Rust engine runs 524k context on one RX 7900 XTX, 5x faster than llama.cpp
W61k3r · reddit · 2026-11-22
Developer W61k3r benchmarked his Rust inference engine LexiSuite running Inkling-2-Mini on a single 24GB RX 7900 XTX: 2,245 tok/s prompt and 52.4 tok/s generation at 4k context vs llama.cpp's 425/25.4 on the same model file. At 262k context LexiEngine hits 1,146 tok/s prompt while llama.cpp failed to finish in 55 minutes. At 524k tokens: 767 tok/s prompt, 33.7 tok/s generation, with 17.6GB weights + 5GB 4-bit KV cache fitting in 23.7/25.75GB VRAM, 13s load time. Needle retrieval is 10/10 at both 260k and 523k depths; accuracy matches the CPU reference (mean KL 0.0038), with residual error from the 4-bit KV cache, not bugs. The built-in draft head speeds up short/medium contexts (79 vs 43 tok/s at 16k) but slightly slows things at 500k+ depth. 1M context doesn't fit on 24GB.
More from Infra
- Dev ports Inkling 2 Mini to vLLM, hitting ~200 tok/s decode on 2x RTX 3090 vs 50 on llama.cpp — peaster_ · 2026-11-22
- Shorting AI-buildout IPOs on day 3 averaged ~37% returns this year — vaibhavbetter · 2026-11-22
- 425 inference API providers now exist worldwide, mostly vLLM wrappers in a losing price war — vaibhavbetter · 2026-11-22
- Qualcomm nears $2B quarterly auto chip revenue, tipped to lead by 2027 — BenBajarin · 2026-11-22
- Gavin Baker: $2T AI CapEx has produced only ~$120B in revenue so far — josh_bickett · 2026-11-22
- Why a $200 subscription can burn $100k of tokens: it's all about infra utilization — RexDouglass · 2026-11-22