llama.cpp PR adds ClassLRU MoE cache: +6-7% decode speed and 8% fewer uploads vs SLRU
pmttyji · reddit · 2026-11-06
PR #31233 in llama.cpp replaces SLRU with ClassLRU, an LRFU/MQ-like hybrid cache that prioritizes frequently used experts in classes ranked by decaying usage history. With a warm cache it delivers +6% decode on QFN and +7% on GLM with 8% fewer uploads, and enables auto mode for llamamoecache pending the merge of PR #30966.
More from Infra
- Linear token pricing breeds bloated agents — rationalized harnesses could run at ~300k tokens — JoshPurtell · 2026-11-07
- Linear token pricing kills incentives: optimized agents could run at 300k tokens — JoshPurtell · 2026-11-07
- Analyst: agents scaling across enterprises will unleash a storage boom as 80% of unstructured data gets contextualized — BenBajarin · 2026-11-07
- Tokens called a great financial invention as billing decouples from value — JoshPurtell · 2026-11-07
- Nvidia and Apollo reportedly eyeing 130+ old telecom sites with ~1 GW for AI data centers — mark_k · 2026-11-07
- Nvidia hiring reveals push to embed with AI-native companies, not just sell GPUs — AJChadha · 2026-11-07