DatologyAI open-sources Zephon, a data loader with elastic determinism for LLM training
josh_wills · x · 2026-11-21
DatologyAI released Zephon, an open-source (Apache 2.0) data loader for text and multimodal training, targeting the pattern of model teams re-inventing data loading for every project.
- Elastic determinism: resume training across different GPU counts (16 → 8/32/64) while preserving identical global batches, even when the pipeline splits, packs, and reorders samples.
- Less repeated preprocessing: tokenization, packing, and mixture ratios happen inside the training pipeline, avoiding extra full passes over the corpus when tweaking data recipes.
- Single Python pipeline for loading, preprocessing, mixing, and batching; optional extras for hf/cloud/parquet/streaming; ships with a paper and blog post.
More from Infra
- Hands-on: AMD ROCm 10.2 delivers 2.2x token throughput on Qwen3.8-27B — AnushElangovan · 2026-11-21
- OpenVINO: the open-source toolkit for running AI on Intel CPUs, GPUs and NPUs — abhishek__AI · 2026-11-21
- AWS Lambda now supports GPUs via EC2 capacity providers — samgoodwin89 · 2026-11-21
- 27,262 thinking tokens, zero answer: Google auto-billed 23 empty Vertex AI crashes via HTTP 200 — Plenty-Jackfruit-218 · 2026-11-21
- Hy4 Flash reads 1M tokens in 44 min on Mac Studio M3 Ultra, 81 min on DGX Spark — AIFlow_ML · 2026-11-21
- local-ai-registry: one battle-tested local LLM recipe per GPU, with evidence attached — LiTianleli · 2026-11-21