Zero-degradation Strata tuning: fixing layer_split lifts decode ~10-15% and halves CPU expert misses
PathfinderTactician · reddit · 2026-11-22
A Redditor shares months of Strata local-deployment tuning with a zero output-degradation constraint. Key points: use Alibaba-recommended sampling (temp 1.0, topp 0.95, topk 20); run "parallel": 2 — each Train slot costs only 0.4-0.6 GiB VRAM and the second Train reuses the first's KV cache, keeping decode busy while one Train tests; don't trust layersplit auto — measured expert coverage was 78% vs the 97.4% it assumes, and setting layersplit 16 cut CPU expert misses from 11.3% to 7.0% and lifted decode 8-13% across context depths (81.0→91.2 tok/s under 150K); lock experts in RAM with STRATANOLARGEPAGES=1 plus Windows 'Lock pages in memory'; add --prompt-cache-every 0 for full KV cache reuse; and a CUDA gradual slowdown after hours of runtime traces to expert cache filling VRAM (WDDM spilling to system RAM), fixable via custom build.
More from coding & agent
- Walmart and Muse to launch first agent-to-agent agentic commerce integration over MCP — AccBalanced · 2026-11-22
- scientific-toolkit: one MCP server fronting nine repos that refuses to fabricate results — Emergency-Tutor-7 · 2026-11-22
- Cross-provider adversarial review: hedge shared model blind spots — craigbalding · 2026-11-22
- Inference Stack Launches One-Command Self-Hosted LLM Serving with vLLM, LiteLLM and llm-d — AccBalanced · 2026-11-22
- Users complain GPT-5.5 writes bloatware and duplicates functions — yacineMTB · 2026-11-22
- SWE-bench cost analysis: tiny Inkling 2 Mini costs 3-5x more per task than DeepSeek V4.1 Flash — hyudryu · 2026-11-22