Branchy: branched training makes multi-device distributed LLM inference 1.5-2.6x faster
ManuXD32 · reddit · 2026-11-17
Indie developer ManuXD32 shares an experiment targeting the bottleneck of distributed LLM inference: instead of sequential layer splits (where added devices add latency and idle time), he trains models whose middle layers are split into parallel branches — a coordinator runs embeddings/trunk layers, branches compute in parallel, and fusion layers merge results, capping network hops at 2 regardless of device count, with dropout tolerance so a slow or disconnected branch's output is zeroed out and absorbed.
Test results with two 300M TinyStories models across a server, laptop and phone over VPN:
- Quality cost is small: loss 1.255 vs 1.143; a blind LLM judge still prefers the branched model 45% of the time.
- A 0.1ms-per-token delay trick makes output identical to single-device runs.
- Branch split is 1.5-2.6x faster than the usual layer split.
- Splitting isn't worth it when the model fits on one device, but pays off hugely when it doesn't: mmap reading dropped a laptop from 8.9 to 0.67 tok/s; sharing across two devices restored 7 tok/s (layer split) vs 9-11 tok/s (branch split).
- Reshaping pretrained models (LFM3 1.7B, Qwen 0.8B) into branches only partially works; training from scratch is better.
More from Infra
- Researchers Find Undocumented x86 Instructions Allowing User-Space Microcode Writes — sull · 2026-11-17
- Inkling-2-Small completes 911 enterprise tasks/hour on one B300 node, 43% cheaper than DeepSeek — ryanshrout · 2026-11-17
- Blackstone data center SPV raises $300M of $800M target as debt pipeline chokes — edgarpavlovsky · 2026-11-17
- MS outage: latent bug from 4-month-old commit, human-written code, AI helped diagnosis — sebkrier · 2026-11-17
- Inside the training-data business: $2k-5k per problem, batches priced in millions — teortaxesTex · 2026-11-17
- Nvidia's problem: OpenAI is "90 plus percent Nvidia" and burning the most cash — ThereWas · 2026-11-17