Qwen N-gram mechanism enriches tokens using 51B table
antirez · x · 2026-08-30
antirez adds details on the Qwen 3.8 Flash Next N-gram mechanism: it enriches the current token representation at Layer 2 using a 51B table and a gate. Layer 1 processing allows time to fetch the 16 small vectors from SSD via masking, optimizing the pipeline with asynchronous retrieval.
More from Infra
- vLLM serves Beam at 1,769 tok/s on 4x B300 NVFP4, publishes Docker image and recipes — vllm_project · 2026-10-16
- SkyRL ships Day-0 RL support for Beam with +62% LoRA decode throughput in vLLM — vllm_project · 2026-10-16
- DFlash2 delivers up to 3.03x decode-only speedup for Beam across five benchmarks — vllm_project · 2026-10-16
- vLLM's Beam serving kernels cut GB200 forward time from 8.15 to 6.47 ms at TP8 batch 128 — vllm_project · 2026-10-16
- Inside Beam: 8-of-256 expert routing across 52 layers, sliding window plus NoPE attention — vllm_project · 2026-10-16
- vLLM ships Day-0 support for Reflection-Beam-1.0, a 501B-param MoE with 1M context — vllm_project · 2026-10-16