LoRA merge + requantize fix boosts Qwen3.8B speed ~50% with zero syntax fails
yb2698 · x · 2026-11-20
bchap1n shared a reproducible local-model fix: merge the LoRA adapter into weights (llama-export-lora) → hybrid GGUF → requantize to IQ4XS (14.2 GiB) with llama-quantize --allow-requantize, keeping MTP and q80 KV @160K.
A/B results (control sandwich, same prompts): the merged config hits 86.6/85.9 tok/s (standard/coding) with tldrScore 10.0/10.0 and zero syntax fails, versus 57.6 tok/s and 3x syntax failures for the original pi-lora setup — a 50% speedup with better output quality.
More from Infra
- Neon's MCP server now executes SQL as a selected Postgres role, constraining agents via DB grants — JeremyCMorgan · 2026-11-20
- Odd Lots: Why surging GPU rental prices may not be the bullish signal AI needs — AccBalanced · 2026-11-20
- Hacked Linux appliances: root via gdb flipping suid on /bin/env, systemd persistence — cyb3rops · 2026-11-20
- Nscale needs US license to serve ByteDance remote Nvidia GPU access; ByteDance is 73% of revenue — SumitGup · 2026-11-20
- Anthropic IPO Will Expose Compute and Inference Market Economics in Public Filings — AccBalanced · 2026-11-20
- Prefix caching: reusing shared KV cache context to cut LLM prefill compute — goyalshaliniuk · 2026-11-20