Halocline inference engine hits 148-182 tok/s on Qwen-Flash-Next with KLD 0.082
Distinct-Pie2389 · reddit · 2026-11-18
Benchmarks for the Halocline inference engine (single RTX 4090, SM89 only) running Qwen-Flash-Next UD-IQ4XS v3.1: warm decode of 148-182 tok/s, 1.6-1.8x faster than Strata 0.2.10 on the same card and quant. Quality vs BF16 with int8 KV cache: PPL 3.2323, mean KLD 0.08195, top-1 90.45%; fp16 differs by only +0.00035 paired dKLD. Long-context: 36K needle decode/prefill at 123/6,298 tok/s (1.8x/2.6x vs Strata), 137K needle at 121/5,664 tok/s, with 262,144 context support.
More from Infra
- CoreWeave faces $19.05B principal-plus-interest bill by end of 2027 — SumitGup · 2026-11-18
- Sarvam's token factory vs OpenRouter: vertical integration vs aggregation explained — itsOmSarraf_ · 2026-11-18
- DeepSeek-V4.1-Flash hits 1,129 output tokens/sec on four AMD MI355X GPUs — Scobleizer · 2026-11-18
- x402 opens four public working groups: domain discovery, identity, tax, card acceptance — kleffew94 · 2026-11-18
- Chutes partners with Alibaba Cloud to offer GLM-5.3 and DeepSeek V4.1 at 35% off — markjeffrey · 2026-11-18
- AWS Adds AI Data Preparation to SageMaker: Auto-Labeling, Synthetic Data, Quality Checks — AWS ML Blog · 2026-11-18