4x Intel B65 build: raising vLLM chunk size to 8192 boosts prefill ~30% on Qwen3.8-Flash-Next

SamSausages · reddit · 2026-10-23

A Reddit user continues tuning his 4x Intel B65 local inference build with full data published on his personal site:

• Key finding: vLLM's default chunk size (--max-num-batched-tokens) of 1024 is conservative; raising it to 8192 yields 30% faster prefill and TTFT. First-token time on a 261K prompt drops from 190.2s (1024) with chunk 16384 performing best.
• Tradeoff: larger chunks cause pauses under multi-session concurrency (2.89s max at 8192 vs 0.61s at 1024) and reduce concurrent session capacity, but in practice pauses are imperceptible for chat and background agents finish tasks faster end-to-end. He's keeping 8192.
• Setup: Qwen3.8-Flash-Next W4A16, BF16 n-gram embedding table pinned in host RAM, vLLM 0.30.0 with XPU graphs, tensor + expert parallel across 4 cards, MTP-3 speculative decoding, 262K window, up to 16 concurrent requests.
• Bests: 84.5 tok/s decode single-user, 243.2 tok/s aggregate at 4 users, 5,142 tok/s prefill on 32K prompts, and a 1,039,940-token prompt via YaRN ×4.

Original post →

More from Infra

• IFM open-sources RL360: 8,192 concurrent sandboxes and ~1.3M rollouts for agentic RL — rupspace · 2026-10-23
• SpaceX seeks access to 80,000 US cell tower sites; Polymarket puts carrier odds at 29% — Polymarket · 2026-10-23
• Modular open-sources MAX framework to free AI compute from single-vendor lock-in — clattner_llvm · 2026-10-23
• Claude Models on Google Cloud (Anthropic managed) is now generally available — claudeai · 2026-10-23
• Scaling GLM-5.3-Flash training to 1M token context, 64x over Megatron baseline — AndrewLBeam · 2026-10-23
• Perplexity's Portable Computer lands on AMD Ryzen AI Max, running local 27B model — AravSrinivas · 2026-10-23

Infra channel →