Adhiraj Ghosh says task-adaptive batch sampling delivered a 3.33x pretraining compute multiplier
pratyushmaini · x · 2026-07-24
Adhiraj Ghosh’s Summer of Data talk covers task-adaptive data curation and adaptive batch sampling for pretraining.
- The core idea is to diversify training batches by concept/task, improving data coverage instead of sampling naively.
- The talk claims this approach produced a 3.33x compute multiplier in pretraining, implying much better efficiency for the same training budget.
- The post links the full talk and frames it as the fifth session in DatologyAI’s Summer of Data seminar series.
More from Research
- Stanford's PaperFormalizer turns research papers into Lean-verified versions with author in the loop — StanfordUniversity · 2026-10-27
- JockeyBench uses Japanese horse racing to test LLM agents — all betting strategies lose money — Zhongwei Xie · 2026-10-27
- Cohere's Invent-A-Dataset paper: zero-seed synthetic data beats frontier APIs on quality and diversity — idanbeck · 2026-10-27
- Hugging Face TRL makes score centering + masked importance sampling default in async GRPO — _lewtun · 2026-10-27
- AutoResearch agent's weird training curve gets dug into instead of discarded — ZimingLiu11 · 2026-10-27
- EMNLP 2026: sub-billion-parameter LMs show superhuman entity tracking, human-like fleeting memory — GretaTuckute · 2026-10-27