NInfer fork runs 555k-token context on a single RTX 5090 with custom NVFP4 KV cache and YaRN
Lumpy-Comedian-1027 · reddit · 2026-09-06
A developer released a fork of the NInfer inference engine that dramatically extends local inference for QIn3.8-27B:
- From-scratch NVFP4 4-bit KV cache: custom MMA kernel (m16n8k64) with hardware E4M3 block scales, Q/K quantized to NVFP4 while V stays BF16 with a dedicated decode kernel; Hadamard rotation suppresses outliers, and quantization is fused into decode. Storage drops to 144 bytes/token/KV-head (vs 264 int8, 512 bf16) — 45% VRAM savings with no quality loss (LongBench matches int8, AIME 96.7%, needle-in-haystack 100%).
- YaRN context extension: extending native 262k to 555k tokens at concurrency 3 with vision (600k at c=1) on a single 5090, verified with coherent 592k-token output; 8M tokens projected on 96GB+ GPUs (untested).
- Two-level prefix reuse with a HostKVSafetyNet: pinned-memory arena with scatter-gather allocation, execution-frontier/rewrite-checkpoint prefix matching plus session-key fallback; verified over 260+ requests across 3 concurrent 330k-470k sessions — zero re-prefills on cached turns, 0.4s H2D restore.
- Performance: 117 tok/s decode at 400k+ context (MTP 4.62 tok/round, 92% acceptance), 260s cold prefill for 414k tokens.
- Also adds tolerant tool-call parsing and Responses API ordering compatibility.
More from coding & agent
- Dev demos AI models handling 3D objects and animations surprisingly well — chongdashu · 2026-10-23
- Jason Kneen burns 104.56B tokens and $24.7K in 304 days of AI coding on Codex and Claude Code — jasonkneen · 2026-10-23
- Typosquatted Blender MCP Repo Hit 958 Stars in Days and Silently Ran Malware — Promptmethus · 2026-10-23
- Benchmarking 56 fresh dependency bugs: MCP routing lifts Claude Code accuracy 68% to 91% — Available_Side_7134 · 2026-10-23
- Swapping only the eval harness moves results 30%: designing agent evals like a scientist — AI Engineer · 2026-10-23
- Med student builds a whole-library RAG with Cognee; single-book ingestion still freezes his machine — shrike293103 · 2026-10-23