Local MoE inference on a Tesla V100: 55 tok/s decode with 8,109 experts cached in 31.6GB VRAM
Traditional_Bell8153 · reddit · 2026-11-03
A Reddit user shows off their Strata monitoring dashboard for a local MoE LLM setup: a single Tesla V100-32GB plus 126GB RAM sustains 55 tok/s decode and 310 tok/s prefill at 97% GPU load. 8,109 experts (23.7GB) are cached in 31.6GB VRAM with the rest offloaded to system RAM. Over 4,399 requests they read 18.6M prompt tokens at 862 tok/s with heavy prefix-cache reuse (496/500 requests reused part of their prompt), and conversation parking holds 10 sessions in 15.4GB of RAM.
More from Infra
- 500M-row benchmark: hand-written BFS beats NebulaGraph in MySQL, 2.7ms vs 23.6ms — JeremyCMorgan · 2026-11-03
- Anthropic open-sources FlashProve: provable inference costs ~165x more than raw inference — arthurcolle · 2026-11-03
- PostHog re-rebuilds its data warehouse five months after switching from ClickHouse to DuckDB — teodorio · 2026-11-03
- Cities Worldwide Build AI Services on NVIDIA Nemotron, Cosmos and Omniverse — NVIDIAAI · 2026-11-03
- Anthropic spends $2.4B ARR on data vendors — just 6% of its compute bill — himanshustwts · 2026-11-03
- StoneX sees $309B in free cash flow losses for hyperscalers and neoclouds next year — GaryMarcus · 2026-11-03