Josh Wills launches data pipeline tool with elastic determinism and JSONL/Parquet support
josh_wills · x · 2026-10-07
Josh Wills announced his new data tooling with two headline features:
- Elastic determinism: pipelines that mix, tokenize, and pack data online can flexibly change the number of GPUs mid-run — a capability inspired by Dbrx Mosaic's Streaming, with the scale-out implementation credited to MaxiBoether.
- Batteries-included support for open formats: JSONL, MDS, LitData and Parquet, with vortexdata, LanceDB and possibly DuckDB on the roadmap for training workloads.
Related event: Josh Wills Open-Sources New Data Loading Library at Datology(5 posts)→
More from Infra
- Windows demo routes coding tasks to local model with GPU spinning, llama.cpp lands on Windows ML — ryanshrout · 2026-10-08
- exe.dev explains crossing the hyper-thread boundary: core scheduling cookies for VM isolation — davidcrawshaw · 2026-10-08
- NVIDIA's LoGRA cuts RL training memory by up to 45.7%, trains 27B model where Adam OOMs — mark_k · 2026-10-08
- Qwen3.8-Flash-Next on 6x3090 without NVLink: prefill 8-10x faster, long-context decode 2-3x — flynth92 · 2026-10-08
- Google DeepMind's Philipp Schmid: Give Every AI Agent Its Own Managed Cloud Sandbox — AI Engineer · 2026-10-08
- GitHub goes down; engineers confirm a fix is in progress — iannuttall · 2026-10-08