smolcluster: distributed LLM training (DDP, FSDP, ZeRO) from scratch in plain Python sockets

East-Muffin-6472 · reddit · 2026-11-21

A developer open-sourced smolcluster, an educational project implementing core distributed training/inference algorithms from scratch using only basic Python and sockets: DDP, Pipeline Parallelism, ZeRO, Expert Parallelism, FSDP and Elastic Data Parallel. The library clusters heterogeneous local hardware — Mac minis, Jetson Nanos, Raspberry Pis, phones/tablets — for local AI training and inference. Still in development, education-focused; ideal for learners who want full control over every detail of distributed training.

Original post →

More from coding & agent

coding & agent channel →