How one Redditor squeezed Raon-1-mini from 50 to 232 tok/s on a 16GB RTX 5070 Ti
McNiiby · reddit · 2026-10-29
A Reddit user with only an RTX 5070 Ti (16GB VRAM) documented pushing KRAFTON's Raon-1-mini-preview (29B MoE, 4B active) decode speed from 50 to 232 tok/s at roughly Q4KM quality.
Key steps:
- Quantization: 3-bit IQ3KS fully in VRAM beat 4-bit options that didn't fit or RAM-offloaded experts (130 tok/s cap), validated with KLD against the reference distribution.
- Vocab pruning: dropping 54K CJK tokens from the 260K vocab was nearly lossless (KLD 0.0931 vs 0.0923) and exposed a sampler scanning the full list each token; fixing it cut sampling from 1.3ms to 0.3ms (15% faster).
- Why ikllama.cpp was initially slower: SITU activation disabled CUDA graphs, and mHC layers caused 1,500 tiny kernel launches per token; the GPU used only 20% of memory bandwidth.
- Kernel work, each A/B tested with KLD: SITU+mHC fusion +31%, small-matmul kernel +12%, fused delta attention +9%, MoE megakernel +7%, dense matvec merging +13%, pipelined megakernel +5%, split long-context attention +2-3% at 16K, and more.
The author released the IQ3KS GGUF on Hugging Face and a fork of ikllama.cpp with build/run instructions (needs 16GB card) — an excellent engineering writeup on running MoE models locally on consumer GPUs.
More from coding & agent
- Dev post-trains Qwen3-4B into a tool-using agent: 66 MB llama.cpp LoRA runs on a Mac, val 0.51→0.83 — DesignerIsland5524 · 2026-10-29
- Dev builds read-only Grok Bot assistant that answers questions from your Mac screen and call history — bfrench · 2026-10-29
- Dev recalls 5,000 GitHub contributions in one day: "I could see reasoning traces in my sleep" — vincent_koc · 2026-10-29
- Open-source diagram-design skill turns plain-text descriptions into polished SVG/PNG diagrams — bfrench · 2026-10-29
- MCP tool definitions can eat 190K tokens: unloading one server cut context from 200K to 10K — shoumikgoswami · 2026-10-29
- 1M URL fetches for ~$900: third-party scraping APIs beat custom pipelines — gaganghotra_ · 2026-10-29