2026-09-15
Kirchenbauer et al. watermark LLM sampling by boosting a hashed green list; on OPT-1.3B, 200-token spans at δ=2 hit 99.4% TPR with 0 observed false positives.
Large language models already write news, code, and homework. Once the text leaves the API, platforms and schools have a weak way to tell whether a machine produced it. Post-hoc classifiers track model quality in the wrong direction: detectors that worked on GPT-2 fall behind GPT-3, and adversarial rewrites punch through them.
The other route is to bury a signal at sampling time. The mark should be invisible to readers, cheap to test, and independent of model weights or the generation API. Low-entropy text is the hard case. After "Barack," the next token is almost always "Obama"; changing it wrecks the sentence.
Kirchenbauer, Geiping, and colleagues at the University of Maryland split the vocabulary with a pseudorandom partition. A hash of the previous token seeds a generator that carves a green list G and a red list R. A hard rule forbids red-list tokens. That version is easy to analyze and fails on low-entropy positions.
The soft rule adds a constant δ to green-list logits, then samples from the softmax. Where many tokens are comparably likely, δ pushes mass onto the green list. Where one logit dominates, the bias barely moves the next-token choice. Detection does not need the model: recompute the same hash, count green-list tokens, and run a one-proportion z-test. With green-list fraction γ and length T, z = (|s|G − γT) / sqrt(Tγ(1−γ)). The paper rejects at z>4, a one-sided false-positive rate of about 3×10^{-5}.
Sensitivity tracks entropy. Spike entropy measures whether the next-token distribution is pinned on a few tokens, and a theorem lower-bounds the expected green-list count. For γ=0.5 and δ=ln 2, that bound is (2/3)T S.
Experiments use OPT-1.3B with prompts from the RealNewsLike slice of C4. Completions are about 200 tokens, about 500 sequences per setting. Quality is oracle perplexity from OPT-2.7B.
| sampling | δ | γ | TPR (z=4) | FNR |
| multinomial | 1.0 | 0.50 | 0.767 | 0.233 |
| multinomial | 2.0 | 0.50 | 0.984 | 0.016 |
| multinomial | 2.0 | 0.25 | 0.994 | 0.006 |
| multinomial | 5.0 | 0.25 | 1.000 | 0.000 |
| 8-beam | 2.0 | 0.25 | 0.994 | 0.006 |
| 8-beam | 5.0 | 0.25 | 1.000 | 0.000 |
At most one false positive showed up in any listed run, so the table records FPR as 0.0. A small green list (γ=0.1) sits on the better Pareto front of z-score versus perplexity. Eight-beam search pulls the curve nearly vertical: stronger marks with almost no extra PPL. With γ=0.25, δ=2, and 8-beam search, mean z-score exceeds 5 by about 35 tokens.
A T5-Large span-replacement attack drops AUC from 0.998 to 0.988 at budget ε=0.1, and to 0.954 at ε=0.3, while raising attacked PPL by about 15 points, roughly 3× the original. Scrubbing the mark wrecks quality first.
Failure cases are mostly low-entropy regurgitation. The model copies human training text, so there is no room to plant a mark.
This became a main line of LLM watermarking: no weight change, open detection, interpretable p-values. The detector depends only on γ and the hash, not on δ. The same test can sit behind different strengths for prose versus code, or switch on only for suspicious users.
For platform audits, homework checks, and filtering synthetic pretraining data, this is more deployable than a classifier that must keep up with the next model. It only works if the provider puts the bias in the sampler.
Low-entropy and memorized text will not trip the test; code, formulas, and templated replies are likely blind spots. The study stops at OPT-1.3B news continuation. There is no instruction model and no multi-turn chat. The attack is random-position T5 replacement. An adversary who knows the hash can flip red-list tokens on purpose. The authors' own calculation: flipping 200 of 1000 tokens still leaves z around 6.3. A leaked key makes that much easier.
If the public hash looks only at the previous token, a synonym swap can scramble the next green list as well. Streaming detection, and spotting a short watermarked span inside a long human document, are left open.