LMSYS demystifies TITO: why token sequence mismatch kills agentic RL runs
srush_nlp · x · 2026-10-28
- LMSYS explains the Token-In-Token-Out (TITO) principle in its Miles framework: the trainer must evaluate the exact token sequence the inference engine consumed and produced during rollout. Agentic rollouts are chains of model calls, tool outputs, harness messages and resumed generations — any mismatch breaks RL training.
- Why it matters: one sample per task for training efficiency, and mathematical correctness via maintaining on-policyness.
- Three failure modes: detokenize-retokenize mismatch, reasoning pruned by chat templates, and lossy chat-template re-rendering.
- Implementation: inference session server, append-only guarantees at three levels, a pluggable TITO tokenizer, and a token-sequence comparator; srush also notes a Lean-verified Rust agent pipeline prototype replacing Jinja message building.
More from coding & agent
- Codex scans your connectors/MCPs unprompted in the background, and you can't tell — kenziyuliu · 2026-10-28
- UCLA and Tsinghua paper probes how far coding agents can go in visual generation — ucla-cmllab · 2026-10-28
- 'Claude Duet has won': devs say use provider-agnostic harnesses as the top model keeps rotating — p_e_cooper · 2026-10-28
- ArkenOS deploys swarm of autonomous scientists; Sam Altman approves — _AustinCalvert_ · 2026-10-28
- Lucas Meijer demos cloud agent workspace running parallel Opus tasks at AI Tinkers meetup — lucasmeijer · 2026-10-28
- Stanford's 3-year medical AI benchmark review: GPT-5.5 drops to 28% when tasks must succeed 3 runs in a row — jonc101x · 2026-10-28