Poolside details how synthetic data and strict checks helped train a 118B coding model
AI Engineer · youtube · 2026-07-25
Poolside says synthetic data is now a core part of pretraining for coding models
Marah Abdin and Robert McHardy describe how poolside manufactures more code data once high-quality human code runs out, but keeps the generation process difficult enough to teach the model instead of making tasks trivial.
- Their pipeline uses templates plus extra context, then varies phrasing and difficulty so the model sees a broad but useful distribution of tasks.
- Existing data is also transformed into multi-stage outputs: character swaps, plot changes, and single prompts turned into multi-turn chats.
- An orchestrator filters every generation and discards outputs that miss the target.
- For training, they use strict reproducibility checks: two replicas on the same data must produce the same number or the run is killed.
- That discipline exposed several scale bugs, including broken GPUs, a tensor-parallel precision issue, and silently corrupted gradients from a race condition.
- The team says the recipe has held up at scale and produced a 118B-parameter model for agentic coding whose early results already edge out GLM 4.5 Air.
More from coding & agent
- NVIDIA says Nemotron 3 Ultra hit 97.1% on agentic RTL chip-design tasks — NVIDIAAI · 2026-07-27
- Tokyo Agent Forge hackathon shipped production-ready AI agents in one day — DavidBennett__ · 2026-07-27
- Long-running agents will need immutable event logs, this thread argues — sebpaquet · 2026-07-27
- Agentic Data Science in Practice: Agents Write Code but Answer Wrong Questions — hugobowne · 2026-07-27
- A VS Code extension adds Markdown-style highlighting to Alchemy string templates — samgoodwin89 · 2026-07-27
- Claude’s Stripe MCP connector is being called unusable after repeated disconnects — evielync · 2026-07-27