Running a 125B model on 8GB VRAM with Strata: 900 t/s prompt processing, 30 t/s generation

BleynSpecnaz · reddit · 2026-10-26

A Reddit user ran Qwen 3.8 Flash Next (Gyro S quant, 125B) on an RTX 5060 Laptop with 8GB VRAM, 32GB RAM and an SSD using Strata, hitting 900 t/s prompt processing and 30 t/s generation—outperforming several Qwen 3.6 35B fine-tunes he tested. Side-by-side renders show clearly better voxel scenes than the 35B model.

Key config parameters

• --mmap-experts: offload experts that don't fit in memory to SSD
• --kv-resident 20480: split KV cache between VRAM and RAM (minimum value)
• --vram-reserve-mib 500-900: reserve VRAM for other tasks
• --pcie-frac 0.35: control PCIe read fraction, tunable via --calibrate
• STRATARESIDENTHEADROOMGIB=4: reserve RAM for the system; higher on Windows

Lessons learned

• Gyro S quant offers the best quality/speed balance; other quants loop repetitively or struggle with non-English
• Skip MTP: it boosts TG but eats VRAM and hurts PP; worth it only with 10GB+ VRAM
• Run Vision on CPU (1-2s per image) to save substantial VRAM
• With 131K+ context, avoid --parallel (OOM risk); use --conversation-cache instead

Original post →

More from Infra

• Running full DeepSeek V4.1 Flash weights on 4x DGX Spark (~$28k): 1M context at 66 tok/s — ricklamers · 2026-10-26
• Meta files $35B stock and mandatory convertible offering, plus $60B ATM program — RihardJarc · 2026-10-26
• "The grid and your home are the bottleneck": compute buyer goes bullish on ARM chips — TheZachMueller · 2026-10-26
• On-device VLM goes production: GenieX runs Liquid AI's LFM-VL live at PyTorch Conference — PyTorch · 2026-10-26
• a16z: OpenRouter token demand up ~27x in a year, open-weight models now serve 72% — TheZachMueller · 2026-10-26
• Giraffe: Open-Source Baseline Testing for OpenAI-Compatible LLM Endpoints — ImmediateBig8479 · 2026-10-26

Infra channel →