STAPLE adds static sparse memory to pretrained LLMs with 50% less serving energy than MoE
BeidiChen · x · 2026-10-13
STAPLE (arXiv:2610.12599) is a recipe for integrating static sparse memory into pretrained checkpoints without restarting pretraining. On OLMo-2 1B it beats continued dense training by +4.27 HumanEval / +2.80 MBPP / +2.27 GSM8K, with gains persisting on 7B. Serving energy per token stays near dense and is >50% below MoE upcycling in high-throughput regimes. Building on DeepSeek's Engram and concurrent STEM, STAPLE adds embeddings as a residual pathway — framed as an energy-efficient scaling axis for a power-constrained era.
More from Infra
- Noise: a speedrun for pretraining transformers without backprop — madhavsinghal_ · 2026-10-13
- P100 x2 + Qwen3: Vibe-Coded llama.cpp Patches Nearly Double Throughput — Mrinohk · 2026-10-13
- Leaked OpenAI slides: ChatGPT tops 1B weekly users, compute grows 10X every two years — firstadopter · 2026-10-13
- Applied Materials unveils EPIC Center as analyst sees WFE demand soaring — BenBajarin · 2026-10-13
- Cloudflare serves Clef with SGLang: sub-40ms latency on a single H200 — ying11231 · 2026-10-13
- Running Qwen3.8 27B with 262k context, q4_0 KV and vision on 16GB VRAM at 80t/s — Fancy-Snow7 · 2026-10-13