New LoRA kernels lift llama.cpp throughput from 300 to 1400 tok/s on Strix Halo
woct0rdho · reddit · 2026-10-21
woct0rdho shares llama.cpp LoRA performance work: a 35MB Qwen3.8-Flash-Next abliterating LoRA dropped prompt processing from 1500 to 300 t/s on Strix Halo. A general LoRA kernel recovered it to 1000 t/s, and a fused kernel to 1400 t/s. Code is open-source, optimized for Strix Halo but portable, with plans to upstream. LoRAs also work across quant formats without redownloading the model.
More from Infra
- Uncensored DeepSeek V4.1 Flash GGUFs drop refusal rate from 91.4% to 0% with minimal MMLU loss — kalyan_kpl · 2026-10-21
- Specter speeds single-GPU MoE inference with quantized multi-step speculative offloading — Wenfeng Wang · 2026-10-21
- SpaceX reportedly targeting 13 GWh of compute by EOY 2027; Grok bots for businesses in weeks — elonmusk · 2026-10-21
- Meta engineer confirms Google TPUs now run AI inference in Meta production — dotey · 2026-10-21
- OmniGraph Open-Sourced: Git-Style Branches and Merges for AI Agent Memory — techNmak · 2026-10-21
- Running Qwen3.8 for agentic coding on 2x5090: vLLM is the only stack that doesn't trash the cache — TooManyPascals · 2026-10-21