vLLM adds day-0 support for Flint: 500 TPS on a single GB200

_rockt · x · 2026-10-22

vLLM officially announced day-0 support for Flint, the first model from Recursive (Richard Socher's company).

• Flint is a 40B-parameter MoE activating only 2.5B params per token, with sliding-window attention in most layers and 262K-token context
• On a single GB200, a single request decodes at 500 tokens/s even with 100K tokens of context, no speculative decoding needed; ships as NVFP4 (quantization-aware trained) and BF16 checkpoints
• Per Socher, Flint targets fast local coding: 125+ tokens/s on MacBook Pro, 150+ on RTX Spark, built from scratch in under 4 months by their automated research system, the Eureka Machine; open weights on HuggingFace
• The blog gives full day-0 Docker (vllm/vllm-openai:flint) and vllm serve commands; support is merging into vLLM mainline (#62938), reusing the glm47 tool-call/reasoning parsers

Original post →

More from Infra

• Qualcomm's non-phone revenue seen growing 60% next year, driven by auto and embedded — pdamodaran · 2026-10-22
• After PHP advisories, a WebAssembly runtime vendor lays out its security validation plan — MickeySteamboat · 2026-10-22
• Ollama Runs Open Models on NVIDIA Blackwell via Crusoe, Live in Two Weeks — ollama · 2026-10-22
• CUDA's moat quietly eroding as Codex can rewrite kernels for AMD, investor notes — paigeinsf · 2026-10-22
• Claim: every 2nd/3rd-tier inference provider now has a recipe for 90%+ compute margins — zephyr_z9 · 2026-10-22
• NVIDIA keynote: silicon photonics CPO switches hit 102.4 Tb/s, Vera Rubin architecture unveiled — thealexbanks · 2026-10-22

Infra channel →