SGLang's hicache: use an L3 storage cache to keep KV cache alive across local model swaps
TheZachMueller · x · 2026-09-22
When running local models, shutting down a server or swapping models destroys the model-specific KV cache — and switching back means recomputing all prompts. SGLang's hicache offers L1 (GPU), L2 (RAM), and L3 (storage) cache tiers; configuring L3 lets old models' caches persist across swaps, eliminating recomputation.
More from Infra
- AMD Engineers Publish GEMM Optimization Tutorial Blog Inspired by the GEMM Ladder — simran_s_arora · 2026-09-22
- Why Meta hasn't shipped Muse in WhatsApp: not enough hardware for 2B users, says user — zephyr_z9 · 2026-09-22
- Terraform Industries makes high-purity methanol at scale, runs solar-direct electrolyzer under $100/kW — GabGarrett · 2026-09-22
- Deep-dive worklog: optimizing CUDA GEMM from naive kernel to shared-memory tiling — abhijithneil · 2026-09-22
- AMD publishes educational GEMM ladder for Helios GPUs: 432GB HBM4, 23TB/s bandwidth — simran_s_arora · 2026-09-22
- Grok 4.7's 81k output tokens per task more than double Grok 4.6's 36k — ArtificialAnlys · 2026-09-22