tito: the hidden tokenizer bugs breaking RL training, and a Lean-verified fix
VoidAsuka · x · 2026-11-02
Background
A long-standing tito (token-in, token-out) problem plagues reinforcement learning: the transformations between messages, bytes, and tokens are not bijective—decoding and re-encoding tokens may not recover the original sequence, and byte-level edits can propagate tokenization changes in both directions. This is harmless in pretraining/SFT (fixed tokenized representations) but causes a nasty class of training bugs in RL rollouts.
Developments
- @ChonghanLiu initially assumed the issue was already solved by the community and internal teams, but after reading srush's blog realized fixes are incomplete: support code is fragmented across wildly different designs with no real standard. He submitted a PR to verl addressing this.
- srush's post "Lean Verified Tokenizers" builds a formally verified (Lean) tokenization pipeline to mitigate harness bugs. Inspired by No Tokens Left Behind, the Aeneas docs, and GitHub's BPE blog; code is open-sourced at srush/certified-tokenizer.
The core mismatch
Agents communicate via structured tool-call messages, yet message renderers allow arbitrary global Python code (e.g., rewriting prior assistant messages when a user message is added, stripping thinking), compounding the inconsistency.
More from coding & agent
- rakyll's AX Studio prototype manages Gemini/OpenAI/Claude agent tasks — rakyll · 2026-11-02
- Claude built an app that auto-downloads Gradescope exams and runs IRT analysis — DataJunkie · 2026-11-02
- OpenAI DevDay London hackathon: Codex reset reportedly arriving within hours — paw_lean · 2026-11-02
- As AI Writes More Code Than Humans Can Review, What Understanding Still Matters? — Panel_pl · 2026-11-02
- Live by the Claude, Die by the Claude: Can Junior Devs Survive AI? — bashaZP · 2026-11-02
- Skip soul.md: just tell your agent never to use first person, argues jessfraz — repligate · 2026-11-02