Benchmarking llama.cpp on 5090+3090: ubatch, lazy-mode and MTP tuning for 60k-260k context
Blindax · reddit · 2026-09-22
A Reddit user benchmarked llama.cpp running Qwen3.8-27B and Flash-Next at 60k-260k context on a Ryzen 9800X3D, 128GB RAM, dual RTX 5090+3090 setup, with reusable tuning findings:
- Lazy-mode regression: new builds lazily map the embedding table, dropping Flash-Next prefill from 1220 to 704 t/s with experts in RAM; disable lazy-mode and use load-mode none.
- For RAM-offloaded MoE, ubatch matters more than expert layer count: ubatch 2048→4096 gives +31% (941→1234 t/s), as bigger micro-batches cut PCIe weight transfers per token.
- Keep batch at 3-4× ubatch on a 2-GPU split: batch=ubatch serializes the two GPUs (2702→2040 t/s).
- A lone 5090 beats the split for 27B: Q6 single-card hits 2766 t/s prefill / 100 t/s decode vs 2687/69.5 for a Q8 split; he defaults to single-card 128k.
- At 262k context the compute buffer spills onto the second card — drop ubatch to 1024.
- MTP speculative decoding: draft 3 is the sweet spot (+12% decode, draft 4 <1%); acceptance holds at 68-75% across code and numeric reasoning, with decode speed driven mostly by context length.
More from Infra
- Fighting AI crawler traffic: beyond Turnstile, Cloudflare's AI Labyrinth as an option — fforres · 2026-09-22
- Software moats won't survive RSI — ML infra's value is demand aggregation, says cHHillee — PatrickToulme · 2026-09-22
- Raspberry Pi locks devices to original RAM size, blocking aftermarket memory upgrades — ngxson · 2026-09-22
- fal's H3 Max generates 5 seconds of frontier-quality video in just 3 seconds — gorkem · 2026-09-22
- NVIDIA's EPD Disaggregation Cuts Multimodal TTFT Up to 5x, E2E Latency 7x — dl_weekly · 2026-09-22
- Running MiniMax H3 locally on a 16GB Mac: 8-10s clips in 15-20 minutes — coberholzer · 2026-09-22