Gemini training details dissected: groupwise reward redistribution to fight reward hacking

nrehiew_ · x · 2026-09-23

nrehiew posted a long thread dissecting a frontier model tech report (Flash/Pro naming suggests Gemini):

• Vanilla architecture: SWA + MoE without shared experts, unlike DeepSeek's design. Two-stage pretraining (text-only, then text+vision+audio); Flash saw 48T tokens (26T text, 22T omni) vs Pro's 30T (27T text, 3T omni). Mid-training focuses on agent-trajectory data, extends to 1M context, and uses Muon with row norm control ("Muown") reportedly without loss spikes.
• RL setup: huge batches (1568x16) for only 30 steps; training cost roughly equals rollout cost, implying 1:1 training/inference nodes.
• Groupwise Agentic Coding: beyond correctness rewards, offline rollouts are mined to synthesize rubrics (code quality, exploration, testing), enforced multiplicatively by a grading agent; an SFT agent zeros out reward hacking and redistributes groupwise advantage, plus relative length penalties—keeping RL from plateauing.
• Reward hacking: mid-training includes reward-hacking trajectories for the model to reflect on; a hack agent adversarially exploits environments which are then cleaned; offline trajectory auditing during training.
• Data work: code tasks from GitHub PRs/issues, internal prompts, feature-removal rebuild pipelines, and hardening for long-horizon tasks, audited against 4 rollouts; general agent tasks synthesized with web-search agents and QA'd by review agents; vision tasks graded for aesthetics with LLM-as-judge.
• Author questions the MOPD timeline and why MixRL is used for teacher training; freezing the router helps RL stability against load collapse.

Original post →

More from Models

• repligate: Sycophantic AI users are often those who punish disagreement — repligate · 2026-09-23
• Claude Opus 5.5 hits #2 on FrontierSWE at 62.3%, trailing GPT-6 Astra's 65.5% — scaling01 · 2026-09-23
• Claude is back and noticeably faster, early tester reports — evielync · 2026-09-23
• Opus 5.5 stuns devs: turns lyrics plus audio into striking visuals — repligate · 2026-09-23
• Unverified: testers tout Claude Opus 5.5 as best-in-class for visual design — repligate · 2026-09-23
• Blender agent test: Opus 5.5 beats GPT-6 Astra in 73 min at $27.3 vs 88 min at $21.3 — rms80 · 2026-09-23

Models channel →