Inkling Small 2 fits on 4x RTX Pro 6000 with 7.5M token KV cache at 120 tok/s
LiTianleli · x · 2026-11-14
- Open-weight model Inkling Small 2 now fits on 4× RTX Pro 6000 (96GB Blackwell) or 3× DGX Spark.
- EXL3 3.48bpw quantization supports a 7.5M-token KV cache with 120 tok/s decode.
- Author open-sourced inkling2-exl3: one docker command downloads the 270GB model and runs c1/c4/c8/c16 decode + prefill speed sweeps (median of 3), outputting a markdown table and JSON; built on an exllamav3 v1.6.6 overlay with Triton kernels and TP4 attention / EP4 experts.
- Author says it's very early and expects 3x c1 and 8x c16 speedups within two days.
More from Infra
- Openship: self-hosted deployment platform with built-in CI/CD hits 15.3k GitHub stars — jackedAJ · 2026-11-14
- Run Gemma 4 locally with one Ollama command: 7-10GB, offline, no subscription caps — nikola_mr64990 · 2026-11-14
- MiniMax H3 I2V on a 5090: cu130 torch fix kills OOM, 0.98MP x 20s in ~21 minutes — Realistic-Fennel-190 · 2026-11-14
- Perplexity team teases minimalist LLM trainer built with coding agents, open source soon — BanghuaZ · 2026-11-14
- Serving many fine-tuned variants of a 27B model on one GPU: 4-bit CQ format and 55x smaller adapters — textclf · 2026-11-14
- Musk: Future Starship Flights Will Carry Over 50 Tb/s of Starlink Bandwidth to Orbit — elonmusk · 2026-11-14