176K context at 36 t/s on 8GB VRAM: full expert-cache config shared
campaigner_ · reddit · 2026-10-14
A Reddit user shared a complete llama.cpp-style config for running a large-context MoE model on 8GB VRAM + 32GB RAM: 36 t/s decode, 340 t/s prefill, 176128 context.
Key settings: --expert-cache 2000, --kv q40, --spec 2, expert-adaptive scheduling flags (--adapt-every/--adapt-swaps/--adapt-decay), and a 300 MiB VRAM reserve. The crucial tradeoff: sacrificing MTP (minimal impact) and caching 2000 experts fixed the prefill bottleneck. No tool-call issues or looping; stable through 3 compactions at 192K context (with fewer cached experts).
More from Infra
- Runway partners with Baseten for dedicated enterprise deployment of Gen-4.5 and Aleph 2.0 — runwayml · 2026-10-14
- Data center panel asks 'Are We Screwed?' as Gary Marcus says AI industry may have screwed itself — GaryMarcus · 2026-10-14
- Australia drafts national AI and data center framework for 2027 as foreign money flows in — YvesMulkers · 2026-10-14
- ASML Posts €11.7B Q3 Net Sales, 57.5% Gross Margin — BenBajarin · 2026-10-14
- Repeated-Data Pretraining Study: Smaller Batches Act as Regularizers and Beat More GPUs — Sewon Min · 2026-10-14
- AWS walks through deploying and benchmarking models on SageMaker JumpStart with reusable Python script — AWS ML Blog · 2026-10-14