Parsed, Plausible, Wrong: Output Format Controls Silent Errors in Small-Model Semantic Parsing
Umar Ahmed, Maxim Dunaev, Mohammad Nour Al Awad, Sergey Ivanov, Ivan Khodnenko
cs.CL, cs.LG
2026-11-13
On PIZZA, plain words cut silent errors from 10.3% to 4.5% on Qwen3.5-0.8B and from 17.9% to 10.0% on LFM2.5-350M, using 4.1× fewer tokens than s-expressions on Qwen.
A semantic parser fails in two ways. An illegal string can be retried. A well-formed tree that is not what the customer said goes to the kitchen with no alarm. That silent rate, S, is the deployment number. Exact match, A, stays so the runs can be lined up with older parsers.
A café parser already in service had picked a compact format for speed. A small handwritten test could not separate formats, so the comparison moved to PIZZA, which turns spoken pizza and drink orders into trees and matches them without regard to child order. The pool has 2,456,446 grammar-generated utterances. The 348-sentence dev set and the 1,357-sentence test set were written by people. The human test set is the evidence that counts.
The same tree was written six ways. On Qwen, outputs run from 44.3 tokens down to 10.9. Shorter looked safer before the measurement. Length alone does not help. Ordinary words do.
All six encodings decode back to the same tree on dev, test, and 50,000 training rows. EXR is the native parenthesized form, 44.3 tokens. JSON is the matched-length bracket swap at 44.1. EXR-SHORT renames constructors to single letters, 32.3. TAGGED is flat key=value, 22.2. DSL marks extras and negation with sigils such as + and !, 11.3. WORDS keeps that structure and spells the markers as words, 10.9. The first four are a length ladder. EXR/JSON and DSL/WORDS are matched-length pairs, so length and wording can be pulled apart. Token counts are mean Qwen outputs at the full pool.
On the 1,357 gold trees, WORDS is longer in characters than DSL (43.1 vs 34.9) and shorter in Qwen tokens (10.3 vs 10.8). The tokenizer splits +cheese into three pieces and leaves extra cheese as two. LFM2.5's vocabulary is about four times smaller, so punctuation-heavy formats inflate the most.
The models are Qwen3.5-0.8B and LFM2.5-350M, the latter mostly convolutional. LoRA rank and alpha stay at 16 on every linear layer, for every format. Loss is on the target string only. Training stops by 4,000 steps (64,000 examples), with a check every 100 steps and patience 10. The stop score averages dev exact match, refusal, and correction exact match. Learning rate is frozen after a per-model, per-format pick from 1e-4, 2e-4, and 5e-4, using dev exact match on a 100k pool and three seeds. The decode cap is 1.25 times the longest target in that format, so a correct string is not truncated.
The grid is 300 runs: 6 formats, pools of 1k, 10k, 100k, 1M, and the full pool, five seeds, two models, with refusals and corrections in every run. Past the step cap a larger pool only diversifies the sample. A seed locks the subset, so formats are paired. The analysis was fixed first: mean of five seeds, 10,000 hierarchical bootstrap resamples, Holm correction across the five pool sizes, separation only at adjusted p below 0.05.
PIZZA has no off-menu class, so hehe hehe becomes a pizza. Negatives are 30% of each training set, and anything beyond a pure order is refused whole. Each format has one refusal string that cannot parse as an order. The 614 eval negatives use words absent from training. Spoken repairs are another 10% of real orders, with a quarter of the cue phrases held out. Dev and test stay pure PIZZA, so A and S keep their definitions.
Every test order falls in one of four bins that sum to 100%. A is exact match. S parses to the wrong tree. Dparse does not parse. Drefuse is a real order turned away. S is the quantity to push down.
Full pool, 1,357 human orders, means over five seeds.
| Model | Format | tokens | exact match | silent | unparseable | real refused |
| Qwen3.5-0.8B | EXR | 44.3 | 85.8% | 10.3% | 0.6% | 3.3% |
| Qwen3.5-0.8B | JSON | 44.1 | 81.6% | 14.9% | 0.3% | 3.2% |
| Qwen3.5-0.8B | EXR-SHORT | 32.3 | 75.2% | 17.4% | 3.3% | 4.2% |
| Qwen3.5-0.8B | TAGGED | 22.2 | 85.9% | 11.6% | 0.5% | 2.1% |
| Qwen3.5-0.8B | DSL | 11.3 | 79.7% | 8.3% | 6.7% | 5.4% |
| Qwen3.5-0.8B | WORDS | 10.9 | 87.5% | 4.5% | 4.6% | 3.4% |
| LFM2.5-350M | EXR | 52.4 | 74.0% | 17.9% | 0.2% | 7.8% |
| LFM2.5-350M | JSON | 57.8 | 70.7% | 19.1% | 0.1% | 10.1% |
| LFM2.5-350M | EXR-SHORT | 35.6 | 70.8% | 16.9% | 2.1% | 10.1% |
| LFM2.5-350M | TAGGED | 24.4 | 71.6% | 21.4% | 0.7% | 6.3% |
| LFM2.5-350M | DSL | 11.4 | 72.0% | 13.1% | 5.7% | 9.2% |
| LFM2.5-350M | WORDS | 12.0 | 77.4% | 10.0% | 5.8% | 6.7% |
Published BART-Large scores 78.6% on this test with EXR, the full pool, and no refusal or correction data. Different job, so the number is only a reference.
Shorter is not more accurate. On the four-format ladder, exact match falls as tokens fall in 10 of 10 model-by-size cells, by at most 1.1 points per ten tokens, and only one cell separates. EXR-SHORT only renames constructors and loses 10.6 points on Qwen at every pool size, and 3.2 on LFM2. TAGGED, about half as long as EXR, ties EXR on Qwen.
WORDS beats DSL on exact match in all 10 cells. At the full pool the gap is +7.8 on Qwen (interval +6.1 to +9.5) and +5.4 on LFM2, and 9 cells still separate after Holm. On LFM2, WORDS spends more tokens (12.0 vs 11.4) and still wins, so brevity is not the mechanism. Against EXR the exact-match gaps are +1.8 and +3.4, and no Qwen pool size separates. Silent errors do, at every pool size: 4.5% vs 10.3% on Qwen, a ratio of 2.3 (1.8 to 3.1), and 10.0% vs 17.9% on LFM2, a ratio of 1.8 (1.4 to 2.3). Bracket formats almost always parse, so their mistakes are silent. WORDS leaves 4.6% of Qwen orders unparseable, which a caller can see. Those Qwen outputs are 10.9 tokens against 44.3 for EXR, about a quarter as long.
Negation is 5% of WORDS silent errors on Qwen and 6% on LFM2, against 19% to 34% and 14% to 28% for the other formats. As a share of orders, Qwen's silent negation errors fall from 2.0% with EXR to 0.2% with WORDS. What WORDS still gets silently wrong is mostly line count, 36% of its Qwen silent errors. Invented entities per order are 0.018 vs 0.100 on Qwen and 0.051 vs 0.123 on LFM2.
Held-out negatives are refused 95.9% to 99.0% of the time on Qwen and 97.2% to 98.3% on LFM2. Real orders are refused 2.1% to 5.4% and 6.3% to 10.1%. Of 289 test orders with a greeting, please, or thanks, Qwen WORDS refuses 6.1%, against 2.6% of the other orders. On held-out repair cues the corrected attribute is right about 80% to 88% of the time on Qwen and 72% to 78% on LFM2. The reported cheese failure is fixed in all 60 full-pool runs.
WORDS leads exact match at every pool size. The lead over EXR is widest at 1k (+4.0 Qwen, +7.2 LFM2) and then narrows. The lead over DSL on Qwen does not: +6.7 at 1k, +7.8 at the full pool. Qwen latency is about 29 ms plus 26.5 ms per token, R2 of 1.00. WORDS is 3.7 times faster than EXR on Qwen (317 vs 1171 ms) and 4.4 times faster on LFM2 (186 vs 821). Qwen GPU memory stays between 1704 and 1806 MiB.
If the ticket is cooked without a person reading it, S is the number to watch. On Qwen, WORDS does not separate from EXR on exact match, and TAGGED's 85.9% sits with 87.5%. Silent errors fall from 10.3% to 4.5% at every pool size. Bracket formats fail while still looking like a legal order. The word format turns some of those failures into parse errors and lowers the silent base rate as well.
The change needs no new labels. Latency follows the token count, and memory barely moves. The edit is in how the tree is written.
Whether the gap holds for larger models or other tasks is open, and the paper says so. The evidence is two sub-billion models and one English ordering set.
On Qwen the paired exact-match gap versus EXR is +1.8 points and does not separate; TAGGED does not separate either. The abstract's claim that the word format is the most accurate is a point estimate. What the tests support is the silent-error drop.
Strict refusal drops 2.1% to 10.1% of real orders, mostly polite ones the training grammar never wrote. Negatives and repairs are synthetic, and 40 handwritten probes do not cover the tail of real disfluency. Retraining one LFM2 cell moved exact match by 4.9 points, against a seed standard deviation of 3.0; the Qwen rerun moved 1.5, against 1.9. Constrained decoding was not run. It can erase unparseable strings and cannot repair a legal wrong tree. Part of the WORDS gain is failing in public; forcing those strings back into trees could push S back up. Learning rates were also chosen separately per format, so some of the gap is optimization.