DeepSeek-V4-Flash tops out at 770 tok/s on one B300 in a vLLM batch test
Moreh · reddit · 2026-07-22
A Reddit user is benchmarking DeepSeek-V4-Flash on a single B300 with vLLM 0.25.0 and says throughput is only about 770 output tokens/s at batch size 256.
- They expected several thousand tok/s on a B300 and suspect a configuration issue.
- deepgemmmegamoe fails on a single GPU because it requires expert parallelism, so they fall back to flashinfertrtllm.
- Dropping speculative decoding roughly doubles throughput.
- They suspect sparse MLA attention may be running eagerly without CUDA graphs.
- The post asks for real-world tok/s numbers and working single-GPU MoE backends.
More from Infra
- LLM Serving Metrics Thread: Why TPOT and Uptime Make or Break User Experience — abhijithneil · 2026-09-11
- PlanetScale launches sharded Postgres: 768 servers acting as one, 1PB scale — dhruv2038 · 2026-09-11
- Can a 7900 XTX 24GB run Qwen locally? Reddit seeks ROCm tok/s benchmarks — thenomadexplorerlife · 2026-09-11
- RTK Terminal Compression Cuts Tokens but Leaves Your AI Coding Bill Unchanged — Bartaseth · 2026-09-11
- SF Compute founder: buying compute is 'an absolutely awful experience' right now — IgorCarron · 2026-09-11
- SmolVM open-sources persistent computer infrastructure for agents that outlive chat sessions — aniketmaurya · 2026-09-11