Training recipe revealed: 2T-token multimodal pretraining with GRPO specialist distillation
smolix · x · 2026-11-06
A disclosed recipe for a full-modality LLM: continued pretraining of a text LLM on 2T tokens spanning text, audio and video; post-training with GRPO RL specialists for conversation, timing, delegation and task completion, distilled on-policy into a single student; plus a self-improvement loop running in noisy simulated chats. Write-up and technical details are linked.
More from Multimodal
- Workshop: Pydantic AI agent turns child profiles into personalized Christmas illustrations — Al_Grigor · 2026-11-06
- Seedance 2.5 short film: neighbors fight over leaves, prompt shared — LudovicCreator · 2026-11-06
- OpenAI's Justin Uberti to reveal GPT-Live speech-to-speech architecture at VapiCon 2026 — juberti · 2026-11-06
- Pruna open-sources P-H3-Image: MiniMax-H3 video model turned into 8-step, 7x smaller local image model — cocktailpeanut · 2026-11-06
- Melite open-sources agentic ComfyUI: agents that generate and extend their own harness — data-cypher-000 · 2026-11-06
- HeyGen gets 50% cheaper via Monid, no subscription needed for AI avatar videos — socialwithaayan · 2026-11-06