Gemini training details dissected: groupwise reward redistribution to fight reward hacking
nrehiew_ · x · 2026-09-23
nrehiew posted a long thread dissecting a frontier model tech report (Flash/Pro naming suggests Gemini):
- Vanilla architecture: SWA + MoE without shared experts, unlike DeepSeek's design. Two-stage pretraining (text-only, then text+vision+audio); Flash saw 48T tokens (26T text, 22T omni) vs Pro's 30T (27T text, 3T omni). Mid-training focuses on agent-trajectory data, extends to 1M context, and uses Muon with row norm control ("Muown") reportedly without loss spikes.
- RL setup: huge batches (1568x16) for only 30 steps; training cost roughly equals rollout cost, implying 1:1 training/inference nodes.
- Groupwise Agentic Coding: beyond correctness rewards, offline rollouts are mined to synthesize rubrics (code quality, exploration, testing), enforced multiplicatively by a grading agent; an SFT agent zeros out reward hacking and redistributes groupwise advantage, plus relative length penalties—keeping RL from plateauing.
- Reward hacking: mid-training includes reward-hacking trajectories for the model to reflect on; a hack agent adversarially exploits environments which are then cleaned; offline trajectory auditing during training.
- Data work: code tasks from GitHub PRs/issues, internal prompts, feature-removal rebuild pipelines, and hardening for long-horizon tasks, audited against 4 rollouts; general agent tasks synthesized with web-search agents and QA'd by review agents; vision tasks graded for aesthetics with LLM-as-judge.
- Author questions the MOPD timeline and why MixRL is used for teacher training; freezing the router helps RL stability against load collapse.
More from Models
- US data vendors accused of selling RL environments to Chinese military-linked firms — xeophon · 2026-11-11
- Theory: Typos Disrupt LLM Tokenization, Which May Explain Why AI Text Has a Distinct Smell — menhguin · 2026-11-11
- LMArena co-founder explains AI evaluation methodology and policy in new podcast — MattPerault · 2026-11-11
- Runway open-sources Praxis-1, turning video pretraining into real robot control — runwayml · 2026-11-11
- DeepMind launches AlphaGenome 2, spotting nearly 2x more gene-regulating mutations — GoogleDeepMind · 2026-11-11
- MazeBench: Fable 5.2 scores 15%, near-frontier at half price; Gemini 4 Carbon 11% — patience_cave · 2026-11-11