Datology AI open-sources Zephon, a deterministic on-the-fly data loader built for massive ablations
pratyushmaini · x · 2026-10-07
The Datology AI team, running hundreds of data ablations, found existing dataloaders couldn't reliably support two needs at once:
- Elastic determinism: the same run must give identical evals on 16 vs 8 GPUs
- On-the-fly tokenization: no re-materializing the dataset after every pipeline change
They built and open-sourced Zephon to process data streaming while staying deterministic, and report it has significantly sped up their experiment iteration.
Related event: Datology AI Open-Sources Zephon Dataloader for Elastic Training(2 posts)→
More from Infra
- Mistral's real news isn't Large 4 — it's a new architecture to ship models faster — _AndrewZhao · 2026-10-07
- Josh Wills launches open-source data loading library — a year of commits as a fossil record of agent-assisted work — josh_wills · 2026-10-07
- Design note: an architecture for experimenting with dynamic curricula in data loading — josh_wills · 2026-10-07
- Elastic determinism for shuffling, tokenizing and packing pipelines in data loading — josh_wills · 2026-10-07
- Finland halts work at two Google data center sites over forest clearing concerns — Polymarket · 2026-10-07
- CoreWeave enters India with AdaniConneX, planning 240 MW of AI capacity — ayushthakur0 · 2026-10-07