Study: word-based output encoding cuts small-model silent errors 2.3x
ITMO · hf · 2026-11-17
This study tackles silent errors in small-model semantic parsing: well-formed but wrong trees that downstream systems can't detect.
- On the PIZZA ordering benchmark, two sub-billion models (Qwen3.5-0.8B, LFM2.5-350M) were LoRA fine-tuned across six lossless encodings of the same tree (44.3 down to 10.9 tokens per order).
- 300 runs (5 pool sizes × 5 seeds) with pre-registered hierarchical bootstrap and Holm correction.
- Shortening alone never helps, but writing the target in ordinary words does: 87.5% vs 85.8% exact match on Qwen while emitting 4.1x fewer tokens, and 2.3x/1.8x fewer silent errors on Qwen/LFM2 — gains concentrated in negation handling.
- Models also learn refusal of off-menu input (95.9-99.0%), at a measured cost of refusing 2.1-10.1% of real polite orders.
More from Research
- Unified hand-object model handles 7 tasks, trained on 100K clips mined from unlabeled video — RoozbehMottaghi · 2026-11-17
- Oxford, DeepMind, OpenAI & Anthropic paper argues safe AI isn't the same as good AI — alex_verem · 2026-11-17
- New paper lays out a research agenda for AI's impact on mental health — jacyanthis · 2026-11-17
- Physicist: AI now beats most junior theorists, and researchers should ride the dragon — IgorCarron · 2026-11-17
- "We formalized it in Lean" — researcher questions what formal proofs actually verify — DimitrisPapail · 2026-11-17
- CVPR 2027 Announced for Seattle, June 20-25, 2027 — CSProfKGD · 2026-11-17