Training neural nets inside LLMs: XOR MLP goes 53% to 98.8% in 40 steps on simulated QwenGPUs

zeeshanp_ · x · 2026-11-03

Developer zeeshanp shared the first experiment from the QwenGPU project: training a 2-8-1 MLP on XOR with distributed data parallelism entirely inside LLMs. Forward pass, backward pass, AllReduce, and updates all run as CUDA kernels on the simulated GPUs.

Test accuracy climbs from 53% to 98.8% in 40 steps
Every replica matches a numpy replay at every step
The team also ran four merged Qwen3.5-2B models, one per GB200, and compared all six collectives (AllReduce, ReduceScatter, AllGather, Broadcast, Reduce, SendRecv) against real NCCL on identical int32 inputs — all identical; the same kernels also run over NVLink on GB200s

Original post →

Training neural nets inside LLMs: XOR MLP goes 53% to 98.8% in 40 steps on simulated QwenGPUs

More from Fun