Thinking Machines releases Inkling 2 Small; 4-GPU setup hits 380K KV cache concurrency
LiTianleli · x · 2026-11-16
Thinking Machines announced Inkling 2 Small, scoring 31 on the AA Intelligence Index — a big jump over its predecessor. It's available today for fine-tuning and inference on Tinker, with a full open-weight release planned soon.
Community benchmarks on 4x RTX Pro 6000 (exl3-4bit, 97.5% top agreement with bf16) show:
- 16x concurrency with 380,000 tokens of KV cache (6.46M per slot)
- 224 tok/s decode (c1)
- 8,150 tok/s prefill
Related event: Thinking Machines releases open-weight Inkling Small 2, runs on 4 GPUs(2 posts)→
More from Infra
- Samsung, Micron, SK hynix face RAM antitrust suit over 697% price surge in 18 months — wen_ragnarok · 2026-11-16
- Cynapse: a free open-source tool to fact-check and audit your local LLMs — _AustinCalvert_ · 2026-11-16
- WEKA open-sources kvlab: replay real agent traffic to benchmark KV cache offload — AccBalanced · 2026-11-16
- A shortlist of LLM inference throughput benchmarks: AIPerf, AgentX, llm-inference-bench — AccBalanced · 2026-11-16
- KubeCon's big theme is inference: a full-stack breakdown from DRA GPU slicing to vLLM and llm-d — AccBalanced · 2026-11-16
- Jensen Huang discusses his comeback as Nvidia CEO with allies in FT interview — SnoozeDoggyDog · 2026-11-16