Local LLM prefill is stagnant: 2x speedup possible but accuracy pays the price
Zeeplankton · reddit · 2026-10-15
A Reddit user notes that local LLM optimization is all about decode — Sushi, Splash and friends (author runs QFN at 40 tok/s on an M3 Max) — while prefill speed has stagnated.
- Experiments on QFN with layer/attention propagation and approximated KV cache pushed prefill from 400 to 850 tok/s (2x)
- But needle-in-haystack accuracy degraded badly — a clear speed-for-accuracy trade
- Open question: is anyone working on smarter prefill approaches, or is slow prefill a hardware limit?
Relevant for local long-context use, where prefill dominates time-to-first-token once decode is fast enough.
More from Infra
- 7 System Design Algorithms to Know: Merkle Trees to CAP/PACELC — goyalshaliniuk · 2026-10-15
- MiniCPM-V 4.7 runs a 35B MoE multimodal model on phones with just 3GB memory — nevrekaraishwa2 · 2026-10-15
- Sipeed MaixStation product page goes live for edge AI dev — xlltt · 2026-10-15
- Nvidia Is Now Worth More Than the Entire Indian Stock Market — theteknosaur · 2026-10-15
- Apple Silicon GEMM benchmarks: M6 does fp8 at ~2x fp16, M5 doesn't — Brilliant-Hall1387 · 2026-10-15
- NVIDIA joins EU's AI in Science Enablers, offering compute, training and cloud resources — nordicinst · 2026-10-15