Distributed RL Post-Training Achieved on 14 Macs

erfan_mhi · reddit · 2026-07-16

Pluralis Research shared a distributed RL post-training experiment where 14 consumer-grade Macs across 4 countries handled rollout generation, while a remote B200 managed bf16 gradient updates.

Key Approaches

Rollouts were entirely connected via standard home networks without relying on data center interconnects; one of the machines was the author's own MacBook.
The rollout phase utilized MLX + int8 inference.
To mitigate deviations from cross-version asynchrony, the team designed two mechanisms:
PULSE: Transmits int8 weight deltas instead of full checkpoints, reducing typical transfer size from 9GB to roughly 82MB.
DPPO-style probability gate: Filters out about 0.3% of tokens that drift too far from the training distribution.

Results

They evaluated Stoa on PaperSearchQA, a multi-turn biomedical retrieval task:

cover pass@1 increased from 29% to 63%
search rate improved from 22% to 84%

Conclusion & Next Steps

The author emphasizes that consumer hardware can shoulder the bulk of rollout costs. Combining this Stoa approach with Agoro—which previously performed pre-training on hundreds of consumer GPUs—could enable migrating larger-scale inference and training entirely onto user-owned hardware.

Original post →

Distributed RL Post-Training Achieved on 14 Macs

More from Infra