One RX 7900 XTX runs 524k context at 767 tok/s, 3x faster than llama.cpp
W61k3r · reddit · 2026-11-22
A developer ran Inkling-2-Mini entirely on a single 24 GB RX 7900 XTX using their own Rust inference engine (LexiSuite), achieving a 524,288-token context with 23.7/25.75 GB VRAM in use and a 13-second load time.
Key numbers (vs llama.cpp b12600, same card and file):
- 4k prompt: 2,245 tok/s prefill, 81.7 tok/s generation with the model's draft head (llama.cpp: 425/25.4)
- 16k prompt: 2,148 / 80.4 tok/s
- 262k prompt: 1,146 tok/s prefill (3m48s), 40.2 tok/s gen; llama.cpp didn't finish prefill in 55 minutes
- 524k prompt: 767 tok/s, 33.7 tok/s
Needle retrieval: 10/10 facts recovered at both 260k and 523k tokens.
The draft head is a big win at short/medium depth but slightly slower at 500k+ depth, as its own attention costs more than it saves. Accuracy matches llama.cpp (KL 0.004 vs CPU reference, error mainly from the 4-bit KV cache). The 45% prefill gain since last version came from a new attention kernel for the global-attention layers. 1M context doesn't fit on this card.
Related event: Custom Rust engine runs 524k context on a single RX 7900 XTX(2 posts)→
More from Infra
- An AI bubble fable: from free cash flow to private credit to annuity blowup — Svwang1 · 2026-11-22
- How GPUs organize work: kernel launches, thread blocks and scheduling explained — blaizedsouza · 2026-11-22
- Alibaba's T-Head files for HK IPO at up to $80B, rivaling Baidu's Kunlunxin — 创业邦 · 2026-11-22
- Can llama.cpp RPC span a Mac mini, Mac Studio and dual RTX 3090s to run a 120B model? — ruffus_or · 2026-11-22
- OpenAI's GPT rewrites AMD flagship GPU RTL, gaining up to 14% PPA in 5 months — 新智元 · 2026-11-22
- LBNL report: data center load growth did not drive up residential electricity prices (2019-2025) — Eliv_nurotic · 2026-11-22