Custom Rust engine runs Inkling-2-Mini at 524k context on one 24GB GPU, ~5x faster than llama.cpp
W61k3r · reddit · 2026-11-22
A developer benchmarked their custom Rust/Vulkan engine LexiEngine against llama.cpp, running Inkling-2-Mini at 524,288-token context on a single RX 7900 XTX (24GB):
- At 4k prompts: 2,245 tok/s prompt, 52.4 tok/s generation vs llama.cpp's 425/25.4 — roughly 5x faster; llama.cpp couldn't finish a 262k prompt in 55 minutes while LexiEngine did it in 3m48s
- Model file 17.6GB, experts kept in Thinking Machines' native 4-bit format, rest 8-bit, 4-bit KV cache, everything in VRAM; 524k uses 23.7/25.75GB, 1M won't fit
- Needle retrieval: 10/10 correct at both 262k and 524k depths
- Draft head speeds up short/medium contexts (79 vs 43 tok/s at 16k) but is slower at 524k — reported honestly
- Accuracy vs CPU reference: mean KL 0.0038 vs llama.cpp 0.0037, a tie; the 4-bit cache, not a bug, accounts for the error
More from coding & agent
- Visualizing the attention mechanism for integrating subagents into a main agent — Sauers_ · 2026-11-22
- This Rust terminal agent builds its own tools and self-updates its kernel with near-perfect accuracy — jasonkneen · 2026-11-22
- Yacine: You Can Gauge a Programmer's Skill by How Much They Trust Claude — yacineMTB · 2026-11-22
- RT: Opus Needs Heavy Oversight in Large Codebases and Tends to Bloat Code — yacineMTB · 2026-11-22
- Developer has three AI models plan the same robotics project, then merges the best ideas — doodlestein · 2026-11-22
- Claude Code Code Mode: orchestrating multi-step tool calls via JavaScript for speed and context efficiency — AIFlow_ML · 2026-11-22