Thinking Machines' Inkling 2 Small: 594B/17B MoE, 1M context, 630 tok/s on vLLM day 0
vllm_project · x · 2026-11-13
vLLM announces day-0 support for Thinking Machines' Inkling 2 Small: a 594B-total/17B-active MoE (48 layers, 384 experts top-8) with text+vision input and up to 1M context.
- Architecture upgrades over Inkling: block attention residuals, attention sink, gated attention, ViT image encoder
- Mixed-precision MoE: NVFP4 (layers 0–39) + MXFP8 (layers 40–47), so it fits on a single GB200/GB300 node
- Native DSpark speculator (RNN head, 7 draft tokens/step), acceptance length 3.5, up to 630 tok/s per user at TP4
- Optimizations: kernel fusion, PDL, tuned Gluon attention kernel, sequence parallelism; LoRA and RL (SkyRL) supported
Post includes a ready-to-run vllm serve command with parsers and speculative decoding config.
Related event: Thinking Machines Open-Sources Inkling 2 Small Weights Under Apache 2.0(13 posts)→
More from Infra
- "Singapore" reportedly accounts for 15-20% of NVIDIA revenue, and the jokes write themselves — JensHonack · 2026-11-13
- SGLang Summit agenda: LinkedIn serves ads model at 10K+ QPS under 80ms — DavidBennett__ · 2026-11-13
- Open weights leak which GPU trained them — six Chinese models read as NVIDIA Blackwell — Antique_Atmosphere97 · 2026-11-13
- SGLang ships Day-0 support for Inkling 2 Small: 576 tok/s per user on Vera Rubin — LiTianleli · 2026-11-13
- Meme explains CUDA perfectly: optimized for many Alice Huangs, not one — djcows · 2026-11-13
- Perplexity's Mac app shows a local models option, hinting at offline capability — dnlkwk · 2026-11-13