The Functionalizer: Lossless Functional Decomposition for Subword Tokenization
Connor Makowski, Willem Guter
cs.CL
2026-07-08
PUA opcodes factor casing, diacritics, and repeats before BPE. Exhaustive vocab shrinks up to 19.7%; a 98M GPT-2 lifts Python parse success from 7.70% to 9.12%.
Subword vocabularies hit a fork on casing and diacritics. Treat hello, Hello, HELLO, and Héllo as four words, and the table bloats; a gradient on Hello barely helps hello. Lowercase and strip accents, and the table shrinks, but the information is gone for good.
The Functionalizer takes a third path: peel surface variation into reversible operators and keep one canonical root. The analogy is an instruction set. A CPU does not ship ADD1 and ADD2 as separate opcodes; it factors the operation from the immediate. Here the operand is the lowercased root. The opcode lives in the Unicode Private Use Area (PUA) and says where to capitalize, which combining mark to attach, or how many times to repeat a character.
The transform runs after a Llama-style regex split, so each piece has a local coordinate frame and position indices stay in 0–255, one byte. Encoding extracts combining marks, records uppercase indices, strips marks, lowercases, and prepends the operator prefix. Decoding applies operators in reverse and claims a bijection. Mixed-case repeats such as Abcabcabc are left uncollapsed.
Current operators:
Default splitoperators=true emits opcodes as standalone prefix tokens, so every casing variant shares the root embedding, at the cost of longer sequences. Fusion can be left on, letting BPE merge frequent cased words. Downstream runs use the fully split setting.
Prefix order, not suffix, is a bet on multi-subword words: the opcode sits in front, so every later piece can still see the formatting bits in attention.
Tokenizer runs set the BPE target to 4096k and merge until exhaustion, measuring how many slots it takes to cover the corpus. Up to 100k documents per set.
| Corpus | Baseline vocab | Functionalizer | Drop |
| Wikitext | 106,023 | 90,531 | −14.61% |
| Python-Codes | 68,471 | 57,012 | −16.74% |
| FineWeb-Edu | 1,214,684 | 975,169 | −19.72% |
| GitHub-Code-Python | 4,071,598 | 3,356,761 | −17.56% |
Mean drop is 17.16%. chars/token also falls 12.88%–17.73% because standalone prefixes lengthen sequences. This is an exhaustion-coverage number, not a 32k/128k production vocab comparison.
Downstream models are GPT-2 Small at about 98M parameters (12 layers, 768-d, 16k vocab, context 512), trained 50k steps on FineWeb-Edu and GitHub-Code-Python, five seeds.
Character perplexity is tied on prose (2.2656 vs 2.2662) and lower on Python (1.5328 vs 1.5697). Sequence overhead at 16k vocab is +8.6% on prose and +18.8% on code. With a fixed step budget, Functionalizer therefore saw about 8%–16% fewer raw bytes.
Greedy generation on 1,000 prompts: Python ast.parse success is 9.12% vs 7.70%, an 18.4% relative lift, with tighter seed variance. Duplicate n-grams fall from 66.0% to 55.8% on prose and 25.5% to 17.9% on code. The other side is ugly. Empty sequences on prose rise from 0.1% to 7.1%, collapse from 70.3% to 78.4%, and valid characters before collapse drop from 357.9 to 217.7. The small model sometimes emits opcode-only prefixes with no root.
This is a pre-tokenizer that fits for an existing BPE pipeline. No new architecture, no learned codebook. The vocab runs show that casing and whitespace repetition really do consume slots. On code, REPEAT turns indentation into "the same space times n", which is closer to structure than the irregular whitespace chunks BPE usually learns.
Gains at 98M are incremental and paid for in sequence length. A production setting would more likely fuse frequent cased words and keep the long tail decomposed. That switch already exists in the framework and is not evaluated. Code and 100M checkpoints are public.
The authors list four. Scale is 98M and 50k steps. Longer sequences mean 8%–16% fewer bytes at a matched step count, so representation gains are tangled with the training budget. Fusion thresholds are unevaluated. Positions cap at 255, diacritics at 13 Latin combining marks; no non-Latin scripts, no ALLCAPS, no lemmatization. Downstream numbers are a bundle of CAPITALIZE + diacritics + REPEAT, so casing versus indentation cannot be separated.
Separately, the 19.7% vocab figure comes from unconstrained exhaustion and is not a 32k-vocab result. Absolute syntax success stays below 10%. Empty sequences and earlier collapse on prose are a real decoding tax at this size. There is no downstream bake-off against TokenMonster or InCa/InDia, only a methodological contrast.