Kimi-K2.6 INT4/NVFP4 local inference benchmarked on 8x RTX Pro 6000 Blackwell
TheZachMueller · x · 2026-10-14
- User keennay published throughput, prefill, and decode data for Kimi-K2.6 running INT4 and NVIDIA NVFP4 quantization on 8x RTX Pro 6000 Blackwell (768GB VRAM) via vllm.
- The new LocalMaxxing llm-inference-bench integration shows matrix-style runs across context lengths (8K-256K) and concurrency levels (1-128) on one page.
- INT4: prefill drops from 5,801 tok/s at 8K context to 4,141 tok/s at 123K; single-stream decode 63 tok/s (8K) to 43 tok/s (123K); 285 tok/s total at C8/8K. NVFP4 hits 360 tok/s total at C16 and supports up to 256K context.
- Author plans to revisit with NVIDIA Dynamo's AIPerf benchmarking tool.
More from Infra
- 'Please don't go into debt': AI devs warn against taking loans to buy GPUs — TheZachMueller · 2026-10-14
- Don't rush to stockpile GPUs for AI: premature lock-in costs more than hardware, dev advises — lxfater · 2026-10-14
- Chainguard's Mythos harness uncovers vLLM cross-request output bug, vLLM says thanks — vllm_project · 2026-10-14
- Dev buys a dedicated $1,899 GMKtec mini PC just to run his AI agents — jarrodwatts · 2026-10-14
- 12 KV caching techniques to optimize LLM inference, from sliding window to MLA — blaizedsouza · 2026-10-14
- Prefix Sliding paper: training-free 3x speedup for long reasoning via prefix + sliding window — cephaloform · 2026-10-14