One RTX 4090 runs Qwen3.8-Flash-Next: a hand-built inference engine, fully gated
Distinct-Pie2389 · reddit · 2026-11-18
What it is
Reddit user Dist-Pie2389 open-sourced Halocline, a fork of Basalt (itself descended from Strata) hardcoded for a single machine: RTX 4090 (sm89, 24GB, 350W), running only the Qwen family.
- Model: Qwen3.8-Flash-Next, Unsloth UD-IQ4XS quant (93.7GB; 125B params, 6B active, 512 experts, 262K native context)
- Memory: 18.2GiB of experts in VRAM, 38GiB page-locked in RAM, nothing on SSD
- Stripped everything the card can't run (DeepSeek path, multi-GPU tiers, FP4 MoE) and ported from Windows-only to Linux
How changes are gated
- Per-function SASS hash comparison after every build: 101 objects, 2,724 functions byte-identical
- 86 CPU + 122 GPU + 1,079 Python server tests at each step
- 900/900 greedy tokens identical to the pre-change build
- KLD gate against a frozen BF16 base (wikitext-2, 100×512 tokens), bases published on HuggingFace
Key results
KV format: int8 wins — fp16's confidence interval includes zero but halves 131K prefill; k8v4/q40 show significant KLD degradation. int8 prefill: 6,019 t/s at 131K.
Draft window: 8 is optimal (266 t/s structured); window 12 adds a step for nothing; window 5 costs 11% on structured output.
More from Infra
- Google placed furthest on vision in Gartner's 2026 Cloud DB Magic Quadrant — rseroter · 2026-11-18
- Cerebras CEO: 75% of data centers will be outside the US by 2027 after midterm results — johncoogan · 2026-11-18
- CoreWeave faces $19.05B principal-plus-interest bill by end of 2027 — SumitGup · 2026-11-18
- Halocline inference engine hits 148-182 tok/s on Qwen-Flash-Next with KLD 0.082 — Distinct-Pie2389 · 2026-11-18
- Sarvam's token factory vs OpenRouter: vertical integration vs aggregation explained — itsOmSarraf_ · 2026-11-18
- DeepSeek-V4.1-Flash hits 1,129 output tokens/sec on four AMD MI355X GPUs — Scobleizer · 2026-11-18