Baseten runs DeepSeek v4.1 Flash at ~2,500 tok/s on Vera Rubin, 1.7x faster than GB300
baseten · x · 2026-11-19
Baseten shared new low-latency inference results for DeepSeek v4.1 Flash on NVIDIA's Vera Rubin platform:
- 2,500 tok/s at batch size 1 with TP4, just 1.9ms per step
- Up to 1.7x faster than their previous GB300 numbers
- Working with SGLang (sglproject) and radixark to upstream the optimizations into open source
- Tri Dao and stel4th presented details at the SGLang summit
NVIDIA's AI Infra account publicly congratulated the teams, making this one of the earliest notable low-latency benchmarks on Vera Rubin.
More from Infra
- Polymarket launches Anthropic IPO perps, ANT trades near $104 ahead of Nasdaq debut — Polymarket · 2026-11-19
- Cloudflare launches Spend Management: auto-pause Workers billing at limits — JeremyCMorgan · 2026-11-19
- One Amazon EBS gp3 command scales a 10M-vector Weaviate index 26×, from 9.8 to 260 QPS — philipvollet · 2026-11-19
- Researcher begs for spare compute as university cluster stalls, ~18 GPU-hours left to finish — vxnuaj · 2026-11-19
- Local MiniMax H3 on 6GB VRAM: 50 minutes tuned vs 6.5 hours on defaults — No_Cow26 · 2026-11-19
- OVH launches Olala mini 7A1B: open-source European 7B LLM that runs on CPU — jedisct1 · 2026-11-19