Running 262K context on a 5090+4070 rig: three Strata patches hit 130 tok/s decode
Fz1zz · reddit · 2026-10-03
A Reddit user details running Qwen3.8-Flash-Next (ISTA GSQ-RCO IQ3XXS) at full 262K context on a RTX 5090 + RTX 4070 Ti SUPER (chipset PCIe x1, 0.8 GB/s) with only 31GB RAM, using the Strata engine to stream MoE experts from an mmap'd GGUF.
- Baseline: stock v0.1.33 with a 36-layer split gave 890 tok/s on 80K prompts; switching between two chats took 20-48s
- Three self-written patches:
- All prompt prefill on the big card (port of Strata PR #269), then copy only used cells to the small card — conversation parking drops to 0.6-1.2s
- Under memory pressure MADVWILLNEED readahead fails, causing millions of major page faults; per-slice pread cuts faults per benchmark run from 24M to 14K
- Bigger prefill chunks (--prefill auto:32768) cut chunk count 4x, avoiding repeated expert re-streaming
Final config (int8 KV, 32K cells/layer, MTP spec 4, vision on): 80K prompt at 1796 tok/s (45s), median decode 128-134 tok/s, follow-up turn on an 80K conversation in 6s; 40K needle and two-conversation parking tests pass. Patches, install script and benchmarks are open-sourced: github.com/ExTV/strata-5090-4070.
More from Infra
- Investor: market hates uncertainty more than money after Nvidia earnings — MartinGTobias · 2026-11-19
- SGLang ships Day-0 serving support for new model DaisyRose-27B — ying11231 · 2026-11-19
- Linux LTS patch flood: 3318 patches for 6.6 alone, 'no way humans are reviewing this' — pkqzy888 · 2026-11-19
- DaisyRose-27B's in-house sandbox: ~80% cheaper than cloud providers, 99.95% success rate — BanghuaZ · 2026-11-19
- Tencent's 220B Hy4-flash quantized to 49 GB weights, runnable on 64 GB VRAM — gladkos · 2026-11-19
- Berth: a lightweight open-source GPU resource manager with multi-GPU scheduling and VRAM parking — Distinct-Pie2389 · 2026-11-19