Running a 35B MoE at ~60 tok/s on a 12GB RTX 4070 Ti: full config
Seraphym87 · reddit · 2026-08-20
A hands-on test of Ornith-1.5-35B-A3B (Qwen3.5 MoE, 36B total / 3B active, Q4KM 20GB) on an RTX 4070 Ti (12GB) + i9-13900KF + 32GB DDR5 via llama.cpp.
The trick: keep as many active experts in VRAM as possible and offload the rest to system RAM — at --n-cpu-moe 28 about 1GB remains for KV cache. Results: 50–56 tok/s sustained generation, 650–700 tok/s prefill, with MTP speculative decoding acceptance around 42–48%. llama-bench shows 64.9/64.3/62.6 t/s generation at n-cpu-moe 26/27/28 while prefill degrades with more offload. Full launch command included (-ngl 99, q80 KV cache, flash attention, MTP draft), proving consumer GPUs can run 35B MoE reasoners smoothly.
More from Infra
- Animation puts AI data center water usage in context amid debate — ATTlKA · 2026-08-20
- Local LLM quantization guide: Hardware thresholds for FP8, NVFP4, and more — Ill_Dragonfruit_3547 · 2026-08-20
- Mojo integrated with MLIR stack, running matrix multiplication on Corsair in days — clattner_llvm · 2026-08-20
- Blueprint raises $1M+ pre-seed led by a16z to speed up hardware iteration — Scobleizer · 2026-08-20
- Mac can now run a 27B model locally that codes, reasons, and sees — TheMoonMidas · 2026-08-20
- RTX PRO 6000 Blackwell Max-Q Review: Ideal for Multi-GPU Towers — TheZachMueller · 2026-08-20