CUHK's block distillation brings block attention close to full attention, cuts 64k TTFT by ~3.1s
量子位 · wechat · 2026-10-20
A CUHK-led team proposes SemanticSeg, an automatic segmenter, and BlockDistillation, a training framework, to fix the two bottlenecks of block attention: how to split text and how to train models for it. The paper was accepted to NeurIPS 2026.
Problem: Block attention splits input into independent blocks with only the last block doing global aggregation, letting KV states be reused across prompts. But heuristic splitting breaks semantics, training-free block attention loses accuracy, and block fine-tuning is expensive and generalizes poorly.
Method:
- The segmenter uses Qwen3-4B-Instruct with a classification head to score candidate split points, with granularity controlled by probability threshold and recursion depth.
- BlockDistillation uses a frozen full-attention teacher to guide a block-attention student, taking 25,859.9ms per step versus 34,941.1ms for block fine-tuning, about 26% faster.
- Three components: block sink tokens mitigate abnormal block-head attention; block dropout gives training signal from more blocks; token-level loss weighting focuses on sensitive tokens.
Results: Block fine-tuning fails to generalize (32.82 vs 40.40 full attention), while block distillation approaches or exceeds full attention. TTFT drops from 509.2ms to 451.3ms at 8k and from 6971.3ms to 3821.6ms at 64k, saving about 3.1s. Ablations show all four components are necessary.
Outlook: The paper discusses integration with Loop Transformers, arguing Loop architectures shift the bottleneck to prefill where block attention's KV reuse helps, though this is speculation rather than measured. Use cases include code assistants (cache hit rate from 0.13 to 0.99) and multi-turn agent workflows.
More from Infra
- IFM open-sources RL360: 8,192 concurrent sandboxes and ~1.3M rollouts for agentic RL — rupspace · 2026-10-23
- SpaceX seeks access to 80,000 US cell tower sites; Polymarket puts carrier odds at 29% — Polymarket · 2026-10-23
- Modular open-sources MAX framework to free AI compute from single-vendor lock-in — clattner_llvm · 2026-10-23
- Claude Models on Google Cloud (Anthropic managed) is now generally available — claudeai · 2026-10-23
- Scaling GLM-5.3-Flash training to 1M token context, 64x over Megatron baseline — AndrewLBeam · 2026-10-23
- Perplexity's Portable Computer lands on AMD Ryzen AI Max, running local 27B model — AravSrinivas · 2026-10-23