Custom LLM engine for one RTX 4090: 1.5-1.8x faster decode with byte-level verification gates
Distinct-Pie2389 · reddit · 2026-11-18
A community member forked the Strata/Basalt engine lineage into "Halocline," targeting exactly one machine (RTX 4090 24GB + i9-14900K) running Qwen3.8-Flash-Next (Unsloth IQ4XS GGUF). He removed unsupported code (DeepSeek-V4.1, multi-GPU paths), ported the Ada fork to Linux, added Qwen's BF16 MTP draft layer, and gated every change with SASS hashes, 207 CPU/GPU tests, 900/900 greedy-token identity, and KLD-vs-BF16 quality gates. Result vs Strata 0.2.10 on the same card and quant: decode 148-182 vs 95-101 tok/s, 36K needle decode 123 vs 68 tok/s, prefill 6298 vs 2403 t/s. His thesis: the future of LLM hosting is per-hardware engine customization.
More from Infra
- Apple hit by massive outage: App Store, iCloud, Apple Pay all down, offices offline — evilsocket · 2026-11-18
- Together AI talks with Ricursive Intelligence on building a frontier local coding model — togethercompute · 2026-11-18
- TensorScale rebuilds AI stack for generative world models, hires across 5 roles — TerryTangYuan · 2026-11-18
- Ex-Meta founder raises $45M for ComputeDesk to tackle compute risk — arnie_hacker · 2026-11-18
- Microsoft opens Web IQ to all Azure customers as a search engine for AI agents — JordiRib1 · 2026-11-18
- Microsoft unveils four custom Azure chips: Maia 300, Boost DPU, integrated HSM and Cobalt 200 — testingcatalog · 2026-11-18