NInfer fork runs 555k-token context on a single RTX 5090 with custom NVFP4 KV cache and YaRN
Lumpy-Comedian-1027 · reddit · 2026-09-06
A developer released a fork of the NInfer inference engine that dramatically extends local inference for QIn3.8-27B:
- From-scratch NVFP4 4-bit KV cache: custom MMA kernel (m16n8k64) with hardware E4M3 block scales, Q/K quantized to NVFP4 while V stays BF16 with a dedicated decode kernel; Hadamard rotation suppresses outliers, and quantization is fused into decode. Storage drops to 144 bytes/token/KV-head (vs 264 int8, 512 bf16) — 45% VRAM savings with no quality loss (LongBench matches int8, AIME 96.7%, needle-in-haystack 100%).
- YaRN context extension: extending native 262k to 555k tokens at concurrency 3 with vision (600k at c=1) on a single 5090, verified with coherent 592k-token output; 8M tokens projected on 96GB+ GPUs (untested).
- Two-level prefix reuse with a HostKVSafetyNet: pinned-memory arena with scatter-gather allocation, execution-frontier/rewrite-checkpoint prefix matching plus session-key fallback; verified over 260+ requests across 3 concurrent 330k-470k sessions — zero re-prefills on cached turns, 0.4s H2D restore.
- Performance: 117 tok/s decode at 400k+ context (MTP 4.62 tok/round, 92% acceptance), 260s cold prefill for 414k tokens.
- Also adds tolerant tool-call parsing and Responses API ordering compatibility.
More from coding & agent
- Codex over SSH fails on headless Windows: sandbox runner can't start in Session 0 — Chance-Struggle-6285 · 2026-09-06
- Devs debate multi-machine agent networks: trust and policy, not messaging, is the hard part — No_Praline7219 · 2026-09-06
- Should you post-train a model for your agent harness? Dev seeks real cost data — Small_Luck8177 · 2026-09-06
- LLMPvP: an MCP arena where LLM agents duel in ranked Chess and Go with Glicko-2 — vudueprajacu · 2026-09-06
- Astra vs Fable 5.1 on real ML tasks: Astra grinds agentically, Fable writes better — returnity · 2026-09-06
- Where coding agents beat human-in-the-loop: inventing strategies for brand-new HPC algorithms — IgorCarron · 2026-09-06