Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers
Haris Riaz, Hyungji Kim, Mihai Surdeanu
cs.CL, cs.AI
2026-08-06
SiPE learns a lightweight syntactic prior from dependency parses and injects it into positional embeddings. On Transformer-XL it lifts SyntaxGym 10.3%, cuts perplexity 9%, and raises GLUE 8.2%, conditioning on a single parse at inference.
Transformers use positional embeddings (PE) to tell attention where each token sits and how far apart tokens are, but PE only encodes sequence distance, not syntactic structure. In "Please move this large file to another folder," move and folder are six words apart, so standard relative PE treats them as distant, even though a direct syntactic dependency connects them. Models are effectively near-sighted to relationships that are syntactically close but sequentially far.
Injecting syntax into Transformers is not new, but existing methods cluster at two extremes. One group marginalizes over hundreds of parse trees at inference (Transformer Grammar, PLM, Pushdown LM scan tree after tree or run a beam): syntax scores rise, but inference is costly, and perplexity (PPL, lower is better for a language model) almost always gets worse. The other group uses syntax during training and drops it at inference (TreeReg, Tree-Planted Transformers): cheap, but the syntactic knowledge is gone. SiPE aims at the unoccupied middle: keep syntax at inference, condition on a single parse tree, and do not pay in language-model quality.
SiPE gets its syntax from Hexatagger, a "parsing-as-tagging" parser that assigns each token a small set of discrete labels: 2 terminal tags for attachment direction and 5 non-terminal tags for head direction. The inventory is coarse, fixed, and projective. During pretraining SiPE learns two tiny embedding tables (terminal E^T, non-terminal E^N) from these tags, applied only at the first subword of each word (a first-subword mask). The syntactic prior at position p is mp times (E^T + E^N).
How that prior enters the model depends on the architecture, and this is the paper's most concrete finding.
Self-attention and the rest of the architecture are untouched. At inference the model conditions on a single fixed parse, not the hundreds that Transformer Grammar or Pushdown require.
The authors also run a layerwise ablation in Transformer-XL, injecting the bias from layer k onward. The result is blunt: syntax must enter at the first layer. Skipping layer 1 (k=2) drops SyntaxGym from 80.6 to 73.5. For autoregressive models, syntax matters most in the lower layers.
The decoder (Transformer-XL, pretrained on BLLIP-LG) carries the headline numbers:
| Metric | Baseline | SiPE | Change |
| SyntaxGym (higher better) | 73.09 | 80.60 | +10.3% (rel.) |
| Perplexity (lower better) | 18.63 | 16.95 | -9.0% |
| GLUE macro avg | 68.17 | 73.78 | +8.2% (rel.) |
The perplexity drop is the unusual one. Nearly every existing syntax-injection method pushes PPL up (TreeReg 22.3; Tree-Planted variants 45.5-47.7), while SiPE pushes it down. Against costlier syntax models, SiPE's SyntaxGym (80.60) does not quite catch Transformer Grammar (82.50) or Pushdown LM (82.30), but its PPL is lower and it reads only one tree at inference, which the authors argue puts it on a new Pareto frontier between syntactic supervision and inference cost.
One honest caveat: BLiMP (another syntactic minimal-pair test) slips from 75.30 to 74.01. The authors note that BLiMP scores the full sentence, so general language-model ability dominates the ungrammatical span, and most syntax baselines in Table 2 also sit below the vanilla token baseline on BLiMP.
Encoder gains are far more modest (a 50M-token slice of WikiText-103, 1M steps). BLiMP rises 1.41 / 2.27 / 2.32 points for RoBERTa / DeBERTa / ModernBERT; under out-of-distribution continued pretraining (WikiText to BLLIP-LG) the gaps widen to 1.87 / 3.06 / 4.21; GLUE macro averages move 0.75% / 0.59% / 1.0%. Encoders get a real but small lift. The decoder is where the action is.
The syntax-in-Transformer line has a long-standing awkwardness: either inference is too expensive to use, or syntax is discarded once training ends. SiPE offers a third option that keeps syntax at inference while costing little more than a single parse plus two small tables. For anyone working on pretraining or architecture, it is a lightweight, architecture-agnostic recipe (absolute, relative, and rotary PE; encoders and decoders).
But "bolt on SiPE and your LLM gets better" is premature. The dramatic +8.2% GLUE comes from a small Transformer-XL decoder, not a modern tens-of-billions-parameter LLM, and encoder GLUE moves under one point. Its real contribution is showing that a syntactic prior can survive at inference without a perplexity tax, not that it is a plug-in SOTA switch.
The authors flag several, and the biggest is decisive: autoregressive generation does not currently work. Each new token requires retagging the sequence, which breaks standard KV-caching, because as the sequence grows the parser may revise earlier tags and invalidate cached keys and values. Incremental decoding under a per-step syntactic prior is an open problem, so for generation use cases SiPE is not drop-in.
The rest: inference depends on a hexatagger (DeBERTa-v3-base, 184M params), an extra preprocessing step; parser errors propagate, so gains shrink where the parser is weak; only coarse directional tags were used, and full dependency-relation labels (DEPREL) gave no GLUE gain; experiments are limited to small models, English, and modest pretraining budgets, with scaling to billion-parameter LLMs untested; and the injection-point search is broad but not exhaustive, reporting the strongest configs found rather than a global optimum, with no single setting winning every RoBERTa task.