srush builds Lean-verified Rust tokenizer pipeline to fix train-inference mismatch that kills RL runs
srush_nlp · x · 2026-10-28
Sasha Rush (srush) published 'Lean Verified Tokenizers', a blog and open-source prototype (srush/certified-tokenizer) formally verifying agent tokenization with Lean. Key points:
- Agents communicate via structured messages (system/user/assistant/tool), but conversions between messages, bytes, and tokens are not bijective — decoding and re-encoding need not recover the original token sequence.
- This mismatch is harmless in pretraining/SFT with fixed token streams, but fatal in RL, where inference alternates with real tools — yet harnesses still build messages with Jinja templates.
- Message renderers historically allowed arbitrary global Python code (e.g., re-rendering all past assistant messages to strip thinking blocks), making behavior even less predictable.
- The proposed fix: a Lean-verified Rust agent tokenizer pipeline, inspired by No Tokens Left Behind and GitHub's BPE writeup.
Related event: srush pushes Lean-verified tokenizers and formal methods for LLMs(2 posts)→
More from coding & agent
- Ben Horowitz: Databricks' agent system of record lifts accuracy from 50% to 84.5% — irinarish · 2026-10-28
- After auditing his own C code with AI, dev no longer trusts unaudited C/C++ — burny_tech · 2026-10-28
- Snowflake Doubles Down on Agentic RL: Overfit First, Then Scale to 65.7 on SWE-bench — StasBekman · 2026-10-28
- Hook-level agent security wins: Microsoft, Anthropic and OpenAI all shifted from gateways to runtime hooks — jacob_posel · 2026-10-28
- Parsing Claude's Raw SSE Stream into a Forensic Timeline of Agent Tool Calls and Diffs — Additional-Ladder-24 · 2026-10-28
- Supabase launches Compute in private alpha: run AI agents and long-running jobs next to your Postgres data — dshukertjr · 2026-10-28