Persisting KV Cache on Free ARM: 15x Faster LLM Prefill
Annual_Manner_5901 · reddit · 2026-08-03
Testing on a free Oracle ARM box (4 cores), the author found that CPU inference is bottlenecked by the prefill phase. By persisting the KV cache to disk, a new process can inherit the prefill, slashing time from 54.4s to 3.5s (or 0.1s from page cache) for 3356 tokens.
Bugs & Failed Attempts
- Discovered that speculative decoding silently broke warm-ahead caching, rendering it useless.
- Halving active experts during MoE prefill sped up by 44% but silently corrupted the KV cache, hurting accuracy.
Surprising Wins
- Rewriting input as label: value format drastically reduced tokens, improved TTFT, and increased fact extraction accuracy.
- Trimming the vocabulary from 151k to 32k entries achieved 17.8% faster decode bit-for-bit losslessly. The project is MIT licensed.
More from Infra
- Engineer's Reminder: Serve Models at Their Original Training Precision — andrew_n_carr · 2026-08-03
- Local Deployment Deep-Dive: Impact of KV Cache Precision on DeepSeek Models — esw123 · 2026-08-03
- OpenAI Said to Discuss $250B Nvidia Backstop for 10GW Data Center — Beth_Kindig · 2026-08-03
- Why DeepSeek's API Is So Cheap: Tiny Model Size Boosts Single-Chip Throughput — AravSrinivas · 2026-08-03
- Compute Squeeze May Force Neo-Labs to Open-Source Frontier Models — gorkem · 2026-08-03
- TensorSharp Benchmark: Speculative Decoding Doubles DeepSeek Speed — fuzhongkai · 2026-08-03