Debugging llama.cpp: sliding-window context checkpoints silently eat your RAM
jomi-se · reddit · 2026-10-31
The author hit a crash running Gemma 4 12B in llama.cpp and traced it to an OOM in RAM, not VRAM. The culprit was context checkpoints piling up during stress testing: checkpoints are snapshots of Sliding Window values at strategic positions, letting you resume from a spot already outside the window without recomputing all token KV values.
The default keeps 32 checkpoints per slot at 320MiB each—too much for 16GB of RAM—while simple chat continuation rarely needs more than 4. He also had Opus generate an animated explainer and wrote a blog post about it.
More from Infra
- Altman says RAM belongs in data centers, not your PC, as OpenAI's buying spree squeezes memory supply — tomwarren · 2026-10-31
- Particle-1, a token-efficient model, launches on Cloudflare AI Gateway — garrytan · 2026-10-31
- Cracks emerge in the AI spending boom as debt mounts and investors waver — TechCrunch AI · 2026-10-31
- Alphabet plunges 12.6% on $51.4B capex as AI spending cracks the market — pdamodaran · 2026-10-31
- Strix Halo one-command Docker for vLLM: Qwen3.8-Flash-Next hits 43 tok/s decode — asfbrz96 · 2026-10-31
- AMD reportedly plans another Radeon GPU and GDDR6 price hike in November — chemist_slime · 2026-10-31