Custom LLM engine for one RTX 4090: 1.5-1.8x faster decode with byte-level verification gates

Distinct-Pie2389 · reddit · 2026-11-18

A community member forked the Strata/Basalt engine lineage into "Halocline," targeting exactly one machine (RTX 4090 24GB + i9-14900K) running Qwen3.8-Flash-Next (Unsloth IQ4XS GGUF). He removed unsupported code (DeepSeek-V4.1, multi-GPU paths), ported the Ada fork to Linux, added Qwen's BF16 MTP draft layer, and gated every change with SASS hashes, 207 CPU/GPU tests, 900/900 greedy-token identity, and KLD-vs-BF16 quality gates. Result vs Strata 0.2.10 on the same card and quant: decode 148-182 vs 95-101 tok/s, 36K needle decode 123 vs 68 tok/s, prefill 6298 vs 2403 t/s. His thesis: the future of LLM hosting is per-hardware engine customization.

Original post →

More from Infra

Infra channel →