How DatologyAI Generated 12 Trillion Synthetic Tokens — And Fixed 4 Pipeline Bottlenecks
AI Engineer · youtube · 2026-10-03
DatologyAI CTO Bogdan Gaza shares engineering lessons from generating 12 trillion synthetic tokens (web, math, code) for frontier pre-training, since the web offers only 30T usable tokens.
- BeyondWeb: seeded rephrasing beats generating synthetic data from scratch.
- Stack moved from split Slurm/Kubernetes to unified Ray, KubeRay and vLLM on EKS on HyperPod.
- Four bottlenecks and fixes: batched S3 metadata fetches (11 days → 2 hours), right-sized partitions plus checkpointing for GPU failures, cross-cluster CPU/GPU scheduling, and vLLM flag sweeps yielding 40% more inference throughput.
More from Infra
- Gridline launches one-API fine-tuning, distillation and batch inference for open models — Scobleizer · 2026-11-19
- Ascend output could hit ~20M chips next year; memory and packaging, not logic, are the bottleneck — teortaxesTex · 2026-11-19
- AMD Loom boosts Qwen3.5-4B prefill up to 2.6x on Radeon 890M laptops — jfowers_amd · 2026-11-19
- Dev submits PR adding periodic checkpoints to GKE sandboxes so agents don't lose progress — yoimnotkesku · 2026-11-19
- AWS Lambda Adds GPU Support via Managed Instances for Inference and Media Workloads — DavidWells · 2026-11-19
- Nvidia Allegedly Made GPUs Artificially Scarce to Keep Prices Sky-High — GaryMarcus · 2026-11-19