1Cat with expert caching nearly doubles decode speed vs Strata: 140 tok/s on 4x Tesla V100
Acceptable_Bell_1791 · reddit · 2026-11-16
A Reddit user benchmarked local inference of a Qwen3.8-Flash-Next UD-IQ3XXS quant on 4x 16GB Tesla V100s (PCIe 3, EPYC 7543, 8-channel DDR4), comparing Strata and forks against 1Cat, which recently added expert caching:
- Single-stream decode: 1Cat leads across the board at 125-143 tok/s vs Strata's 75-79
- Prefill: 1Cat dominates short/mid prompts (1,850 vs 861 tok/s at 2.6k); Strata wins at 96.5k tokens (3,384 vs 2,649)
- Strata has slightly better MTP draft acceptance (0.6-0.7) yet still loses; the author notes 1Cat would pull further ahead with better acceptance
- Same 103K prompt run three times: 1Cat decode climbs from 99.9 to 138.4 tok/s (+24%), Strata only +10%
Takeaway: expert caching roughly doubles decode speed for local deployment on older/consumer GPUs.
More from Infra
- OpenAI made ~$13B last year but committed ~$1.15T to compute — Qweskj · 2026-11-16
- Pirateface launches P2P network for sharing open models without centralized hosting — alohahaho · 2026-11-16
- China, not the US, is blocking H200 sales; analyst calls chip policy decade's biggest self-own — firstadopter · 2026-11-16
- Agents need throwaway DB copies, not mocks — copy-on-write cloning may become default infra — petrusenko_max · 2026-11-16
- Free LLM Inference Engineering Study Group: From KV Cache to vLLM — V-on-run4_here · 2026-11-16
- Self-hosted daily game pipeline runs Qwen3.6-35B and Qwen-Image locally on DGX Spark, zero API fees — jbsan · 2026-11-16