Running a 176B Qwen MoE on a 16GB RTX 3080 laptop — 4x faster than Strata end-to-end
fuzhongkai · reddit · 2026-10-04
The author ran Qwen3.8 Flash Next 176B on an RTX 3080 Laptop (16GB VRAM) + 32GB RAM + SSD using TensorSharp, his open-source local inference engine — no 128GB RAM workstation or multi-GPU setup needed.
The approach: quantization + MoE-aware unified scheduling across cache, VRAM, system RAM, and SSD. Instead of treating the SSD as last-resort swap, TensorSharp coordinates memory tiers around MoE execution to keep the right experts/data in the right tier at the right time.
Benchmark vs Strata on the same hardware:
- Decode throughput: 11.09 vs 10.24 tok/s (close)
- End-to-end time: 16.54s vs 62.15s (4x faster)
- Peak VRAM: 14,832 MiB vs 15,729 MiB
Takeaway: for huge sparse MoE models the question isn't "do I have enough RAM/VRAM to fit the model?" but "how efficiently can the runtime coordinate VRAM, RAM, SSD, caching, and expert activation?"
Related event: TensorSharp runs 176B MoE model on a 16GB RTX 3080 laptop(2 posts)→
More from Infra
- Anthropic's compute spend grows 4x/year, $10B in Q3 alone, set to overtake OpenAI by May 2027 — scaling01 · 2026-11-19
- Thailand data center association cites $18B in GPU/hardware cancellations after SE Asia lending halt — SumitGup · 2026-11-19
- Gridline launches one-API fine-tuning, distillation and batch inference for open models — Scobleizer · 2026-11-19
- Ascend output could hit ~20M chips next year; memory and packaging, not logic, are the bottleneck — teortaxesTex · 2026-11-19
- AMD Loom boosts Qwen3.5-4B prefill up to 2.6x on Radeon 890M laptops — jfowers_amd · 2026-11-19
- Dev submits PR adding periodic checkpoints to GKE sandboxes so agents don't lose progress — yoimnotkesku · 2026-11-19