Packing dataset files into sequential blobs lifted one training pipeline from 36 to 47 steps per minute
irinarish · x · 2026-07-21
- The author sped up an ML training pipeline from **36 steps/min to 47 steps/min** by changing how datasets are stored on disk. - On consumer hardware, storage is often the bottleneck: **NVMe is expensive**, while **SATA HDDs are slow**, especially for datasets made of many small files. - The fix was to **pack files into sequential compressed blobs** such as **TAR, Parquet, or WebDataset** so the drive can do mostly sequential reads instead of random seeks. - If the data is already compressed, storing it **uncompressed** can avoid extra compression/decompression overhead; the author cites **FLAC** as an example. - The post also notes that very large training runs often rely on **network storage blocks in the same datacenter** as the GPUs, because the dataset is too large to keep on device.
More from Infra
- Developer teases more open models and new llama.cpp updates — mervenoyann · 2026-07-21
- TSMC reportedly plans up to 10% chipmaking price hikes in 2027 — kimmonismus · 2026-07-21
- Ramp says its model router cuts LLM costs by 30% by picking the cheapest good model — rohanpaul_ai · 2026-07-21
- More open models and llama.cpp updates are coming, says Merve Noyan — mervenoyann · 2026-07-21
- Why adding a second LLM provider breaks more than the API surface — Ok_Extension6373 · 2026-07-21
- UK AI datacentres face backlash over heat, noise and land use — nordicinst · 2026-07-21