30 tps for a 35B MoE model on an RX 6700 XT — full llama.cpp Vulkan config shared
Loose_Doubt367 · reddit · 2026-10-10
A user shares a complete, working llama.cpp Vulkan configuration that runs the 35B-A3B MoE model at 30 tokens/s on modest hardware (RX 6700 XT 12GB, R5 5600X, 32GB DDR4), asking how to reach 40 tps without changing quantization or the 100k context.
Verified tuning details: threads=6 beats 4/8/12; flash-attn on with q80 K/V cache; n-gpu-layers=99 with n-cpu-moe=28 for GPU/CPU MoE offload; draft-MTP speculative decoding (n-max=3, p-min=0.75); DIO loading with fit-target=256 MiB. The full config is posted for anyone to replicate.
More from Infra
- Cadence announces sovereign AI cloud for European chip designers — karlfreund · 2026-10-23
- SGLang Deep-Dive Blog Series Breaks Down KV Cache, Posts up to 15.5x Throughput Gains — BanghuaZ · 2026-10-23
- DynaLM: Open-Source CPU-First LLM Inference Engine Seeks Feedback — Unlikely_Safety_7456 · 2026-10-23
- Capital Is Now the AI Buildout Bottleneck, Says Analyst Citing Jensen Huang — JOBhakdi · 2026-10-23
- DynaLM: an open-source CPU-first LLM inference engine seeks testers and feedback — Unlikely_Safety_7456 · 2026-10-23
- Cerebras explains how it deploys trillion-parameter models on wafer-scale chips — bookwormengr · 2026-10-23