Reconstructing Closed-Source Tokenizers via Two Oracles
bayes-song · reddit · 2026-07-11
The author poses a question about reverse-engineering closed-source LLM tokenizers: If you can only obtain two oracles from the chat API—
- Token length oracle: Takes a string s as input and returns the token count of tokenize(s);
- Prefix token oracle: Given a string s and an integer n, returns the decoded result of the first n tokens of tokenize(s).
The first is easy to obtain, while the purpose of the second is to resolve merge order ambiguity, such as distinguishing between (a, bc) and (ab, c). The author suggests that this prefix oracle could be constructed from a standard chat interface by leveraging the model's ability to controllably repeat text within a conversation.
Finally, the core question is raised: Relying solely on these two oracles, is it possible to reconstruct an equivalent tokenizer that matches the token boundaries of the original tokenizer on any arbitrary input (allowing for token ID remapping)?
More from Research
- Linear Digressions returns with a new season of audio essays on AI agents — ChrisGPotts · 2026-07-21
- ARISE study tested 45 AI clinical tools in 1,100 consult cases — HealthcareAIGuy · 2026-07-21
- Async OPD distillation doubles throughput while matching synchronous math accuracy — _lewtun · 2026-07-21
- A forecasting lesson on why R-squared alone led to overfitting and worse predictions — mdancho84 · 2026-07-21
- Google DeepMind’s Project Genie talk shows how creatives feed into model research — alexanderchen · 2026-07-21
- Nat Lambert says RL distillation does not use the strongest models as teachers — natolambert · 2026-07-21