Halocline: hand-rolled CUDA engine runs Qwen4 on one RTX 4090, up to 2.6x faster than Strata

Distinct-Pie2389 · reddit · 2026-11-20

Developer T-Crypt open-sourced Halocline, a sparse+dense CUDA inference engine purpose-built for a single RTX 4090 (sm89 only), forked from basalt with goals of beating ninfer+strata, achieving lower KLD, and full development transparency. Supports UD-Q4 quants and above.

Benchmarks vs. Strata 0.2.10 (same card, same quant):

The author also documents the engine's limits honestly: only validated on one machine; the gate scores 512-token chunks so it misses KV rounding deep in long contexts; weight quant and KV format aren't separated (whole 4-bit build KLD 0.0820); dense path loads Q80 ssm tensors as BF16. Ninfer comparisons are being posted to the repo.

Original post →

More from Infra

Infra channel →