Kimi-K3.1-Pro (2.8T) Squeezed From 1.77TB to 775GB, Runs on One GPU With Zero Accuracy Loss
Grawnch · reddit · 2026-11-09
A Redditor compressed the 2.8T-parameter Kimi-K3.1-Pro (natively 1.77TB, normally needing 22x 80GB GPUs) down to 775GB (-56%) and ran all 93 layers on a single RTX PRO 6000 Blackwell + 176GB RAM + NVMe.
Key techniques:
- Exploits MoE skew: routed experts quantized in 3 tiers (2.81/2.56/1.44 bpw, averaging 1.92), while shared experts, attention and embeddings stay BF16/F32
- Strata fork: a 3-tier offloading engine (hot VRAM / warm pinned RAM / cold NVMe mmap) with router-lookahead prefetching, 97.9% effective hit rate
- dr-triton: fused Triton dequant kernels keep experts packed over PCIe, expanding only in GPU SRAM
- Tier splits are configurable; the custom layout won't load in vanilla llama.cpp/vLLM, but the dequant module is importable
Quality is nearly intact: 29/30 (96.67%) on both AIME 2025 and LiveCodeBench, matching the base model problem-for-problem; Top-1 token agreement 95.2%, Top-5 99.9%; perplexity up only 1–2.7%. Throughput: 36.8 batch / 10.8 single-stream TPS on the test rig, with projections from 12.5 TPS (1x4090 + 128GB RAM) up to 58 TPS (2x PRO 6000 + 512GB RAM).
More from Infra
- Jensen's downfall was expected: with Western ASICs rising, no more mercy needed — teortaxesTex · 2026-11-09
- Reflection AI Ships First Model Beam; Employee #1 Recalls a Year Building From 1k H100s to GB300 Clusters — bigblueboo · 2026-11-09
- BofA: Global data center spending tops $1T, projected to reach $2.1T by 2028 — Beth_Kindig · 2026-11-09
- RTX 3090 beats 128GB ASUS Spark and M5 Max on identical MiniMax H3 video benchmarks — Ok_Bird4483 · 2026-11-09
- Satire: Jensen's 'AI safety master plan' — resign, crash Nvidia, delay AGI a decade — ShakeelHashim · 2026-11-09
- Chamath: SpaceX could reprice Anthropic compute deal at 2-3x on 90-day notice — rohanpaul_ai · 2026-11-09