Running a 176B MoE on a 16GB RTX 3080 laptop: TensorSharp beats Strata in end-to-end test
fuzhongkai · reddit · 2026-10-04
The author ran Qwen3.8 Flash Next 176B on a RTX 3080 laptop (16GB VRAM) + 32GB RAM + SSD using their open-source inference engine TensorSharp.
The approach treats SSD not as last-resort swap but coordinates memory/storage tiers around MoE execution via quantization + unified scheduling, keeping the right experts in the right tier at the right time.
Benchmark vs Strata:
- Decode: TensorSharp 11.09 tok/s vs 10.24 tok/s
- Whole-process time: 16.54s vs 62.15s
- GPU peak: 14,832 MiB vs 15,729 MiB
- OS working set: 19.74 GiB vs 18.51 GiB
Takeaway: for huge sparse MoE models the question isn't "does it fit in RAM/VRAM?" but "how efficiently can the runtime coordinate VRAM, RAM, SSD, caching and expert activation?" The author invites comparisons with llama.cpp and Strata on similar hardware.
Related event: TensorSharp runs 176B MoE model on a 16GB RTX 3080 laptop(2 posts)→
More from Infra
- Gridline launches one-API fine-tuning, distillation and batch inference for open models — Scobleizer · 2026-11-19
- Ascend output could hit ~20M chips next year; memory and packaging, not logic, are the bottleneck — teortaxesTex · 2026-11-19
- AMD Loom boosts Qwen3.5-4B prefill up to 2.6x on Radeon 890M laptops — jfowers_amd · 2026-11-19
- Dev submits PR adding periodic checkpoints to GKE sandboxes so agents don't lose progress — yoimnotkesku · 2026-11-19
- AWS Lambda Adds GPU Support via Managed Instances for Inference and Media Workloads — DavidWells · 2026-11-19
- Nvidia Allegedly Made GPUs Artificially Scarce to Keep Prices Sky-High — GaryMarcus · 2026-11-19