Branchy: branched training makes multi-device distributed LLM inference 1.5-2.6x faster

ManuXD32 · reddit · 2026-11-17

Indie developer ManuXD32 shares an experiment targeting the bottleneck of distributed LLM inference: instead of sequential layer splits (where added devices add latency and idle time), he trains models whose middle layers are split into parallel branches — a coordinator runs embeddings/trunk layers, branches compute in parallel, and fusion layers merge results, capping network hops at 2 regardless of device count, with dropout tolerance so a slow or disconnected branch's output is zeroed out and absorbed.

Test results with two 300M TinyStories models across a server, laptop and phone over VPN:

Original post →

More from Infra

Infra channel →