Multi-hour llama.cpp optimization experiments on Qwen MoE: patches, benchmarks, repro guides
Aelexi93 · reddit · 2026-09-18
A developer ran multi-hour llama.cpp optimization experiments (driven by Astra) to speed up local MoE inference while keeping model weights and quantization unchanged, targeting Qwen3.6-35B-A3B and Qwen3.8 Flash-Next. Hardware: RTX 4080 (16GB VRAM), Ryzen 9 5900X, 64GB DDR4 RAM.
Several workloads saw substantial gains, especially prompt processing and source-code editing; regressions and tradeoffs are documented alongside results. Tests include correctness checks but don't establish unchanged quality across every task. Source patches, benchmark summaries, and reproduction guides are on GitHub (Abzolute1/llama-cpp-optimization-lab); the author welcomes independent reproduction on other hardware, and suggests handing the repo to your coding agent. The portable clean-build route still needs end-to-end validation.
More from Infra
- Fire at Taiwan Panasonic PCB materials plant may tighten CCL supply further — zephyr_z9 · 2026-11-04
- Same Claude model served by Bedrock one day and Anthropic the next, dev finds — georgos_pappas · 2026-11-04
- Self-improving software is inevitable: closing the agent-observability loop — bibryam · 2026-11-04
- Gigabyte's AI TOP motherboard packs 2TB RAM and 9 GPUs for home AI labs — vincent_koc · 2026-11-04
- Building a 7-GPU 132GB VRAM local inference rig: consolidate on EPYC or split across two systems? — infectiousstupidity · 2026-11-04
- imec neural probe integrates photonics directly into 130nm CMOS backend — jwt0625 · 2026-11-04