vLLM ships recipe for Thinking Machines' Inkling-2-Small: 594B MoE, 1M context, ~1,100 tok/s/GPU
vllm_project · x · 2026-11-13
The vLLM team published a serving recipe for Thinking Machines Lab's Inkling-2-Small with a single vllm serve command.
Key specs:
- Multimodal MoE: 594B total / 17B active parameters, text+image input, up to 1M token context
- Architecture: 48 layers (40 sliding-window + 8 full attention), relative-bias attention instead of RoPE, block attention residuals replacing the residual stream, 384 routed experts top-8 (NVFP4 in layers 0-39, MXFP8 in 40-47)
- Ships a DSpark speculator (7 tokens/step, probabilistic drafting), 3.5 acceptance length at T=1.0
- Performance: 1,100 output tok/s/GPU at 100 tok/s/user on TP8; agentic multi-turn hits 1,070 output TPGS at 32K context, 700 at 128K
- Accuracy verified via inkling-verify (NIAH up to 1M, MMMU-Pro, HLE); LoRA and RL (SkyRL) supported
Related event: vLLM adds Thinking Machines' Inkling-2-Small with 1M context and 1100 tok/s(2 posts)→
More from Infra
- 2-bit DeepSeek V4.1 Flash runs a coding-agent seat on two 2016 V100s: a 12-hour real-work trial — a4mula · 2026-11-13
- IBM QDC 2026 Unveils Nighthawk r2 Quantum Chip and Fault-Tolerant Dev Tooling — MJBiercuk · 2026-11-13
- Why Jensen Huang Splits From US Frontier Labs on Open Source and Export Controls — morganlai · 2026-11-13
- SGLang-Neuron v0.1.0 open-sourced: SGLang serving stack now runs on AWS Trainium — ying11231 · 2026-11-13
- Used GPU prices on eBay ranked by $/GB: P100 at $10.9/GB, V100 servers nearly doubled in weeks — MzCWzL · 2026-11-13
- Nvidia Has $36B in Off-Balance-Sheet Backstops for Neoclouds, Plus $20B in Leases — SumitGup · 2026-11-13