Elastic determinism for shuffling, tokenizing and packing pipelines in data loading
josh_wills · x · 2026-10-07
Design point #1 of Josh Wills' new data loading library: elastic determinism for online data prep pipelines involving mixing, tokenizing, and packing. Inspired by Databricks Mosaic AI's Streaming, which lets training jobs move to a different GPU count on demand, the library brings the same capability to online data preparation at scale.
Related event: Josh Wills Open-Sources New Data Loading Library at Datology(5 posts)→
More from Infra
- ServeTheHome's 10K-word DGX Station teardown: 800Gbps to Blackwell Ultra but only 229Gbps to Grace — AccBalanced · 2026-10-08
- Apple-style compression: LSP learns which subspaces to drop, cutting LLM weights 70% — Massimo Bini · 2026-10-08
- Apple's Stepped MoE: one model scales 1-4B parameters, beating dense counterparts — apple · 2026-10-08
- Tokens Now Grow in Potato Fields: Ulanqab Hosts Over 100 AI Data Center Projects — pstAsiatech · 2026-10-08
- Hybrid agent pattern: cloud Gemini plans, local Gemma swarm runs 97% of tokens offline — clmt · 2026-10-08
- Nvidia-Backed Lambda Raising Up to $4B at $14.5B Valuation Ahead of 2027 IPO — darian314 · 2026-10-08