Forcing Gemma 4B to output unseen Simple Wikipedia bigrams makes it typo constantly
cephaloform · x · 2026-09-30
Researcher @erinbeess ran a constrained inference experiment on Gemma 4B, only allowing tokens that form bigrams never seen in the Simple Wikipedia corpus.
The model produced lots of typos — presumably because Wikipedia barely contains any, so satisfying the constraint forces it to invent malformed word combinations. A fun demonstration of how strongly LLMs depend on corpus token distributions.
More from Research
- Diffusion Models Tutorial Accepted to NeurIPS 2026 Alongside 7 Paper Acceptances — mittu1204 · 2026-09-30
- AMB3R-SLAM: Kilometer-Scale Real-Time SLAM on One Consumer GPU, Cutting ATE by 70% — rsasaki0109 · 2026-09-30
- Prefix-Reuse FLOPs: new metric exposes hidden cost of arbitrary context edits in LLM serving — RulinShao · 2026-09-30
- Tencent Hunyuan releases ExplorationBench to measure how AI systems explore — TencentHunyuan · 2026-09-30
- Fully open MolmoAct 2 tops independent robotics benchmark LIBERO-MAX on dynamic robustness — DJiafei · 2026-09-30
- CompVis improves Distributional Diffusion Models: 4.48 FID at 4 steps on ImageNet — CompVis · 2026-09-30