Josh Wills Open-Sources New Data Loading Library at Datology
On October 7, engineer Josh Wills published a launch blog post for a new data loading library at Datology and open-sourced the code. Aimed at data preparation for large models, the repo also preserves a full year of commit history, which he described as a "fossil record" of how his team's way of working with agents evolved over the past year, adding that he'll discuss that part in depth later.
Confirmed
- The library's first design point is "elastic determinism" for online data preprocessing pipelines covering shuffling, tokenization, packing, and more, inspired by approaches from Databricks Mosaic's Streaming... work, giving such pipelines scalable yet deterministic behavior
- The library supports common data formats like JSONL/Parquet
- The third design point is architectural support for dynamic curriculum learning experiments
Why it matters
- Josh Wills noted that much interesting research in dynamic curricula never reaches the broader community precisely because there's no well-maintained, well-documented data loading library to handle the drudgery; this library aims to take on that grunt work so researchers can focus on research itself
- The repo's full year of commit history of agent collaboration is itself a valuable primary source on how AI-assisted engineering practice is evolving—worth watching for his follow-up discussion
2026-10-07 ~ 2026-10-07 · 5 related posts
Primary sources
- Josh Wills launches open-source data loading library — a year of commits as a fossil record of agent-assisted work — josh_wills ·
- Elastic determinism for shuffling, tokenizing and packing pipelines in data loading — josh_wills ·
- Design note: an architecture for experimenting with dynamic curricula in data loading — josh_wills ·
- [source] Elastic determinism for shuffling, tokenizing and packing pipelines in data loading — josh_wills · 2026-10-07
- Josh Wills launches data pipeline tool with elastic determinism and JSONL/Parquet support — josh_wills · 2026-10-07
- [source] Josh Wills launches open-source data loading library — a year of commits as a fossil record of agent-assisted work — josh_wills · 2026-10-07
- [source] Design note: an architecture for experimenting with dynamic curricula in data loading — josh_wills · 2026-10-07
- Datology opens year of agent collaboration commits as an RL-able fossil record — josh_wills · 2026-10-07