Multi-Branch Transformer: parallel branches plus Joint-KV beat sequential Transformers
Gaotang Li · hf · 2026-11-18
- Idea: Instead of only widening or deepening Transformers, Multi-Branch Transformer (MBT) runs independently parameterized blocks in parallel behind shared initial layers, fusing outputs through shared final layers.
- Variants: Wide-Shallow, Narrow-Deep, and Balanced designs studied under matched training compute.
- Results: Controlled pretraining across 0.8B–2.1B params at four MoE scales shows 2-branch Wide-Shallow MBTs consistently beat sequential Transformers at matched budgets; quality holds as branches grow from 2 to 4.
- Joint-KV: A shared KV cache built from branch representations, plus a branch-grouping inference engine, improves long-context efficiency.
- Team: UIUC, Rice, Michigan, Cornell.
Related event: Multi-Branch Transformer Scales LLMs via Parallel Branches(3 posts)→
More from Infra
- AMD Loom boosts Qwen3.5-4B prefill up to 2.6x on Radeon 890M laptops — jfowers_amd · 2026-11-19
- Dev submits PR adding periodic checkpoints to GKE sandboxes so agents don't lose progress — yoimnotkesku · 2026-11-19
- AWS Lambda Adds GPU Support via Managed Instances for Inference and Media Workloads — DavidWells · 2026-11-19
- Nvidia Allegedly Made GPUs Artificially Scarce to Keep Prices Sky-High — GaryMarcus · 2026-11-19
- AWS adds private CA support to Bedrock AgentCore Identity for connecting agents to self-hosted IdPs — AWS ML Blog · 2026-11-19
- Leak Hints at Classified 1M-GPU Training Cluster in China That Skips HBM — teortaxesTex · 2026-11-19