Halocline: hand-rolled CUDA engine runs Qwen4 on one RTX 4090, up to 2.6x faster than Strata
Distinct-Pie2389 · reddit · 2026-11-20
Developer T-Crypt open-sourced Halocline, a sparse+dense CUDA inference engine purpose-built for a single RTX 4090 (sm89 only), forked from basalt with goals of beating ninfer+strata, achieving lower KLD, and full development transparency. Supports UD-Q4 quants and above.
Benchmarks vs. Strata 0.2.10 (same card, same quant):
- Warm decode, short prompt: 148–182 tok/s vs. 95–101 (1.6–1.8x)
- 36K needle decode/prefill: 123 / 6,298 tok/s vs. 68 / 2,403 (1.8x / 2.6x)
- 137K needle: 121 / 5,664 vs. 87 / 2,864 (1.4x / 2.0x)
The author also documents the engine's limits honestly: only validated on one machine; the gate scores 512-token chunks so it misses KV rounding deep in long contexts; weight quant and KV format aren't separated (whole 4-bit build KLD 0.0820); dense path loads Q80 ssm tensors as BF16. Ninfer comparisons are being posted to the repo.
More from Infra
- 753B-parameter GLM-5.3 served fully in GPU memory across two NVIDIA DGX Stations — kuchaev · 2026-11-20
- PolyAI's real-time dialog model Dialog-RSN-1 lands on SageMaker JumpStart — AWS ML Blog · 2026-11-20
- Three decommissioned V100s in a $1,042 rig beat a $5,399 Mac Studio at long contexts — bittingthembits · 2026-11-20
- NVIDIA's Rubin debuts in MLPerf Training and Inference as benchmark maintainer congratulates strong quarter — TheKanter · 2026-11-20
- US chip curbs bite: Moonshot drops $150M Indonesia deal, TSMC and Samsung halt China shipments — The Decoder · 2026-11-20
- SemiAnalysis exposé: "Japan's first hyperscale B300 cluster" chaired by ex-Spain PP leader — cross__entropy · 2026-11-20