One RTX 4090 runs Qwen3.8-Flash-Next: a hand-built inference engine, fully gated

Distinct-Pie2389 · reddit · 2026-11-18

What it is

Reddit user Dist-Pie2389 open-sourced Halocline, a fork of Basalt (itself descended from Strata) hardcoded for a single machine: RTX 4090 (sm89, 24GB, 350W), running only the Qwen family.

How changes are gated

Key results

KV format: int8 wins — fp16's confidence interval includes zero but halves 131K prefill; k8v4/q40 show significant KLD degradation. int8 prefill: 6,019 t/s at 131K.

Draft window: 8 is optimal (266 t/s structured); window 12 adds a step for nothing; window 5 costs 11% on structured output.

Related event: Hobbyist Open-Sources Halocline, a Custom Inference Engine for a Single RTX 4090(3 posts)→

Original post →

More from Infra

Infra channel →