Paragraph Boundaries Are Not White Space:Compression Depth as the Signature of Hierarchical Structure
Shuyang Xiang
cs.CL
2026-09-20
Fixing tokens and changing paragraph position compresses attention; Code U*=-0.772 vs random-label -0.238, so depth marks real structure.
Standard positional encodings treat text as a line: the link between tokens i and j is a function of reading-order distance. Paragraphs are nested. The same two sentences, glued into one paragraph or split into two, have identical tokens and identical token distances. A human reader does not treat them as identical. An n-token sequence admits 2^{n-1} paragraph segmentations, so reading order is not a sufficient statistic for hierarchy.
The paper asks a narrow question. Is attention compressed near a paragraph boundary? Is that compression deeper under real structure than under a matched random control? And is the effect caused by the paragraph coordinate rather than by token content?
hRoPE assigns paragraph index p1, sentence-in-paragraph p2, and token-in-sentence p3 to separate rotary channel groups that never collapse into one scalar. That isolation lets the authors change only p1 with the token sequence held fixed.
Two interventions: fake-merge assigns two real paragraphs the same p1; fake-split inserts a fake boundary inside a paragraph. Attention is averaged over layers and heads, then residualized against a log-linear fit on token distance; the residual is Deff. A normalized score β maps the within-paragraph baseline to 0 and the real-boundary mean to 1.
Matched variants: flat (ordinary RoPE), sentaxial (sentence and token axes only), hropeaxial (all three), randaxial (same architecture, p1 resampled each step at matched boundary density), and periodaxial (p1 = floor(t/L), used only as a mirror control). Models are small: 8 layers, 8 heads, dmodel=512, context 1024, 5000 steps, three seeds per corpus. Corpora: WikiText-2, OpenWebText, Python source. Compression depth uses an exact-distance estimator that keeps only (d, Δp1) cells with at least 2,000 token pairs; an earlier binned estimator flipped WikiText-2's sign.
Changing p1 alone changes attention. On WikiText-2, fake-merge ΔB = -1.300, fake-split ΔC = +0.841, and the fake boundary relative to real ones is βC = 1.060. OpenWebText is similar. Code is larger: ΔB = -3.769, ΔC = +3.373, βC = 1.522, so the fake split exceeds the average real boundary. flat and sentaxial, which have no p1 channel, sit near zero.
What occupies the channel matters. At inference, swapping p1 on a trained hrope checkpoint yields validation loss real < period < rand < const0 in every seed of every corpus. On Code, real is 2.383 nats and collapsing p1 to a constant is 2.720, a 0.337-nat gap. Across corpora the constant condition is 0.30-0.34 nats worse than real. With three seeds this is directional (minimum one-sided sign-test p = 0.125), not a formal significance test.
Compression depth U (more negative is deeper): Code -0.772, WikiText-2 -0.481, OpenWebText -0.305. randaxial is compressed too, more shallowly: Code -0.238, WikiText-2 -0.220. On OpenWebText, hrope -0.305 versus rand -0.326, and the paired interval includes zero. Compression itself is therefore not diagnostic of true structure; depth is. The displacement of the well, Δp1, is structurally unidentifiable on WikiText-2 (median paragraph about 122 tokens, so large-Δp1 cells fail the 2,000-pair gate).
Eight corpus-only quantities, grouped into lexical persistence, paragraph length, and embedding-based coherence, none fully recover the Code deeper than WikiText-2 deeper than OpenWebText ordering. Adjacent-paragraph embedding cosine minus a random-pair baseline comes closest (0.096 / 0.066 / 0.051) and separates 2 of 3 corpus pairs. Lexical persistence λstruct swaps WikiText-2 and OpenWebText. Median paragraph length (55 / 122 / 45 tokens) swaps Code and WikiText-2.
Reading-order distance is not structural distance. An extra density-matched axial channel also compresses attention, so seeing compression is not the same as a model that understands paragraphs. If a hierarchical coordinate is worth adding, depth is the stable signature and it depends on the corpus; the mechanism is still open. All runs use 8-layer, 512-wide models trained for 5,000 steps, with no downstream metric. Treat this as a phenomenon paper, not a positional-encoding recipe.
Three corpora and three seeds: the cross-corpus order is descriptive. The real-versus-random depth gap is unresolved on OpenWebText. periodaxial is nearly a function of token distance, so the exact-distance protocol cannot score it. An additive attraction-plus-suppression family is ruled out; softmax competition is not. No production scale, no downstream tasks. The paper does not claim that an ordinary Transformer already stores paragraph coordinates. Paragraphs are defined corpus-specifically (prose blocks versus blank-line code blocks).