LLM autoresearch boosts Qwen3.8 decode speed 2.4x on RTX 5060 Ti in 20 hours
bobaburger · reddit · 2026-11-19
A Reddit user let a cloud LLM run automated research on llama.cpp to speed up Qwen3.8 Flash Next on an RTX 5060 Ti 16GB. Over 20 hours and 13 experiments, decode speed went from 24 to 40.6 tok/s on Unsloth IQ1S, and from 8.4 to 20.1 tok/s (+139%) on IQ4XS — with byte-identical output vs the baseline.
Method: 12 test prompts (coding/math/Q&A), decode-speed-only measurement, byte-to-byte output comparison instead of KLD/PPL, logging speed/cache hit rate/disk reads and keeping only improvements.
What worked: tuning flags (-np, KV cache quant, batch/ubatch), resizing VRAM expert cache (16 GiB pinned host pool + batched prefetch → 19 GiB pool → ODIRECT demand reads at 16 threads), and direct llama.cpp source changes the author admits he can't fully understand (code is open-sourced).
Discarded: ngram speculative decoding (verification touches 20–32 experts per layer and evicts the VRAM cache), CPU offload of misses, lookahead prefetch, CPU thread changes, and requantization attempts (one gained speed but dropped quality).
The author's takeaway: the LLM can autonomously dig into llama.cpp code for optimization clues, but he can't claim full credit since he doesn't understand all the changes.
More from coding & agent
- a16z maps the consumer agent infra stack: search, access, comms, memory, payments — FinanceYF5 · 2026-11-19
- Next.js security advisories jumped to 51 this year as AI digs up old bugs — cramforce · 2026-11-19
- New research report benchmarks strong ranking models solo and combined for multi-stage retrieval pipelines — CShorten30 · 2026-11-19
- Dev argues AI's next step is human-agent collaboration spaces, not full automation — FinanceYF5 · 2026-11-19
- Devs find infinite canvases surprisingly effective for ideating with AI agents — max__drake · 2026-11-19
- Proposal: coding agents should A/B test new models blindly across turns — iamsahaj_xyz · 2026-11-19