Persisting KV Cache on Free ARM: 15x Faster LLM Prefill
Annual_Manner_5901 · reddit · 2026-08-03
Testing on a free Oracle ARM box (4 cores), the author found that CPU inference is bottlenecked by the prefill phase. By persisting the KV cache to disk, a new process can inherit the prefill, slashing time from 54.4s to 3.5s (or 0.1s from page cache) for 3356 tokens.
Bugs & Failed Attempts
- Discovered that speculative decoding silently broke warm-ahead caching, rendering it useless.
- Halving active experts during MoE prefill sped up by 44% but silently corrupted the KV cache, hurting accuracy.
Surprising Wins
- Rewriting input as label: value format drastically reduced tokens, improved TTFT, and increased fact extraction accuracy.
- Trimming the vocabulary from 151k to 32k entries achieved 17.8% faster decode bit-for-bit losslessly. The project is MIT licensed.
More from Infra
- Baidu open-sources FlashMask: up to 1.55x faster than FlashAttention-4 for long-context training — BaiduResearch · 2026-10-16
- GYRO quant of Qwen3.8-Flash-Next tested: near Q3 quality at much smaller size — PlasticRevenue4601 · 2026-10-16
- RL train-sample systems are frontier AI's hardest infra problem — and a trade secret — brianryhuang · 2026-10-16
- Local tuning of GLM-5.3-Flash on 2×GB10 lifts decode from 49 to 58 tok/s — EAccelerate_42 · 2026-10-16
- 3x DGX Spark cluster runs GLM-5.3-Flash locally behind a proxy, 1.75M tokens/day at 119W — Teknium · 2026-10-16
- Optuna creator Akiba: when 2 hyperparameters cost $20M, what's left for BO/HPO — Stefania_druga · 2026-10-16