Help Needed: Tuning llama-server Cache
Reasonable_Goat · reddit · 2026-07-12
A user running multiple models on a Strix Halo machine, especially **Qwen 3.5 122B**, found that cache hits significantly impact performance. A full cache miss at 100k context causes preprocessing time to spike to 10–20 minutes, heavily disrupting agentic workflows. They listed their current configurations, noting that while helpful, several issues remain: - `--cache-ram 16384`: Expands memory available for cache - `--ctx-checkpoints 72` and `--checkpoint-min-step 4096`: Splits context into multiple checkpoints - `--parallel 1`: Gives single session exclusive full cache access - `-fa on, --spec-type draft-mtp --spec-draft-n-max 3`: Author believes this is largely unrelated to caching Remaining pain points include: - Large checkpoint intervals mean crossing a checkpoint during fast agent iterations takes time - Sometimes misses checkpoints after a user turn, especially in certain agent interaction flows - Despite setting 72 checkpoints, older ones seem to gradually disappear, leading to larger recalculations - The author is considering whether K/V quantization (like q8) could retain more checkpoints with less memory, and its impact on quality Ultimately, the post is asking for advice on tuning llama-server cache strategies.
More from Infra
- Fluidstack raises $830M at $7.5B valuation as Anthropic backs a $50B compute buildout — rohanpaul_ai · 2026-07-21
- Early Krea2 Gradio WebUI targets 6GB low-VRAM local runs — Fluid_Kaleidoscope17 · 2026-07-21
- Z.AI starts running a 1GW AI data center built entirely on domestic chips — Polymarket · 2026-07-21
- Local models feel far more capable once paired with the right harness — Soft-Barracuda8655 · 2026-07-21
- Voice-agent teams should use platforms first, then own STT events when failures get weird — FollowingSuitable941 · 2026-07-21
- Alibaba open-sources T-Head SAIL, the software stack behind its AI chips — 量子位 · 2026-07-21