Older Strata silently decodes greedy; ROUTE_TAIL_SKIP=7 halves 4-bit quant quality
Distinct-Pie2389 · reddit · 2026-10-18
While migrating a Strata setup from IQ2XS to Unsloth UD-IQ4XS (RTX 4090, i9-14900K, 64GB DDR5), the author hit two practical issues:
1. Greedy decoding on older builds
Before Strata 0.1.45, a run config without a sampling block decoded greedy whenever the request didn't set a temperature — the cause of looping outputs. 0.1.45 (#2685) now applies model-card defaults (thinking: temp 1.0/topp 0.95/topk 20; non-thinking: temp 0.7/topp 0.8/presence penalty 1.5). Older builds need the config block added manually.
2. Route tail skip trades away quality
Since 0.1.42 the default STRATAROUTETAILSKIP=7 skips low-ranked experts for speed, but Strata's own QUANTS.md shows it doubles the 4-bit file's KL against Q80 (0.053→0.115). The author's A/B on UD-IQ4XS at 262K context: skip 7 gives 93 t/s decode, skip 0 gives 72-81 t/s with identical prefill speed. They chose quality, setting it to 0.
Also: running the official calibrator (tools/calibrate.py, 30 min) bumped --spec-min-p from 0.5 to 0.7 for +4%; anything ≥0.55 for --pcie-frac cost 20-40%.
More from Infra
- Running Qwen Flash Next on a Mac Mini M5 64GB: SSD expert streaming benchmarked with charts — turtleninja99 · 2026-10-18
- Xolotl runs 552B DeepSeek-V4.1-Flash on a single RTX 4090 at ~27 tok/s — indrasmirror · 2026-10-18
- Running agent swarms under 6GB VRAM: millisecond micro-models as workers, LLM only for decisions — everettplays · 2026-10-18
- Rust-based Git server on S3 handles 7,600+ concurrent pushes per second — blaizedsouza · 2026-10-18
- Chaining 2 PCs, a MacBook and 2 iPhones Runs 48-Expert MoE at 37 tok/s on 60k Context — StayLameBro · 2026-10-18
- Ollama US hosting coming soon, bringing DeepSeek-V4.1-Flash-1017 stateside — gaganghotra_ · 2026-10-18