RustingStrata: pure-Rust GGUF engine runs Inkling 2 Mini at 3.5x llama.cpp write speed on an RTX 3060
Serious_Scarcity7818 · reddit · 2026-11-22
A developer added Thinking Machines' Inkling 2 Mini (26B) support to RustingStrata, a pure-Rust GGUF inference engine, and benchmarked it against llama.cpp on an RTX 3060 12GB.
- Unusual architecture: 24 layers, 12 of which run 3x per token (48 layer runs per token); the IQ4NL GGUF is 17.6GB — bigger than the 12GB card
- Speed: RustingStrata CUDA writes at 31.2 tok/s vs 8.7 tok/s for llama.cpp (-ngl 18) on the same card — 3-3.5x faster; prefill reads are 1.5x slower than llama.cpp
- Why writes are fast: the engine keeps the most-used experts in VRAM, computes the rest on CPU, and shifts the split dynamically
- Long context: at a 40,960-token prompt, writes drop to 16.7 tok/s; llama.cpp needs a 13.2GB compute buffer at default batch (won't fit), and 2.86 tok/s with -ub 128
- Correctness check: feeding llama.cpp's greedy tokens through its forward matched 24/24 on short prompts and 47/48 on longer ones, across CUDA/Vulkan/CPU
Supports CUDA, Vulkan and CPU-only with no flags; text-only, greedy only, up to the model's 1M context. Open source on GitHub (RustingStudio/RustingStrata); the author asks for Vulkan numbers from AMD/Intel cards.
More from Infra
- Teknium's Hermes telemetry shows 94% of tokens are cache reads, only 0.6% uncached — Teknium · 2026-11-22
- Teknium shares Hermes telemetry: 92.4% of tokens are cache reads — Teknium · 2026-11-22
- Musk demos real-time AI video generation on half of one AI4 TRIP accelerator — jamesdouma · 2026-11-22
- Sentdex asks for ~150 free H100s right after Martin Shkreli posted wanting to buy them — Sentdex · 2026-11-22
- ECB finds $4.5 trillion in hidden AI debt, 287% above 2025 tally — Actual__Wizard · 2026-11-22
- Anomaly undercuts Fireworks/Baseten by 60% on open model pricing — natolambert · 2026-11-22