Halocline inference engine hits 148-182 tok/s on Qwen-Flash-Next with KLD 0.082

Distinct-Pie2389 · reddit · 2026-11-18

Benchmarks for the Halocline inference engine (single RTX 4090, SM89 only) running Qwen-Flash-Next UD-IQ4XS v3.1: warm decode of 148-182 tok/s, 1.6-1.8x faster than Strata 0.2.10 on the same card and quant. Quality vs BF16 with int8 KV cache: PPL 3.2323, mean KLD 0.08195, top-1 90.45%; fp16 differs by only +0.00035 paired dKLD. Long-context: 36K needle decode/prefill at 123/6,298 tok/s (1.8x/2.6x vs Strata), 137K needle at 121/5,664 tok/s, with 262,144 context support.

Related event: Hobbyist Open-Sources Halocline, a Custom Inference Engine for a Single RTX 4090(3 posts)→

Original post →

More from Infra

Infra channel →