Multi-hour llama.cpp optimization experiments on Qwen MoE: patches, benchmarks, repro guides
Aelexi93 · reddit · 2026-09-18
A developer ran multi-hour llama.cpp optimization experiments (driven by Astra) to speed up local MoE inference while keeping model weights and quantization unchanged, targeting Qwen3.6-35B-A3B and Qwen3.8 Flash-Next. Hardware: RTX 4080 (16GB VRAM), Ryzen 9 5900X, 64GB DDR4 RAM.
Several workloads saw substantial gains, especially prompt processing and source-code editing; regressions and tradeoffs are documented alongside results. Tests include correctness checks but don't establish unchanged quality across every task. Source patches, benchmark summaries, and reproduction guides are on GitHub (Abzolute1/llama-cpp-optimization-lab); the author welcomes independent reproduction on other hardware, and suggests handing the repo to your coding agent. The portable clean-build route still needs end-to-end validation.
More from Infra
- Google engineers: LLM benchmark harnesses silently drop requests — 200 QPS in, 38 out — AI Engineer · 2026-09-20
- The rig built to run Emacs and doomscroll X is now worth more than its owner's car — tetsuoai · 2026-09-19
- Apple M4 sustains 10 instructions per cycle, beating most rivals; M5 speedup explained — lemire · 2026-09-19
- Apple M6 bumps cores to 12 with two super cores; CPUs keep improving fast — lemire · 2026-09-19
- Apple M-series chips gained ~50% Geekbench 6 performance over three years — lemire · 2026-09-19
- Inside OpenAI's inference routing: why the proportional controller had to go — AI Engineer · 2026-09-19