Fractional State Space Transition for Long Sequence Modeling
Ivan Kobyzev, Abbas Ghaddar, Ali Nasiri-Sarvi, Lifeng Shang, Yufei Cui
NeurIPS 2026 (Oral)
cs.LG, cs.CL
2026-09-29
FRAC fits fractional power-law memory with 16 log-spaced exponential modes. At 1.3B, LongBench is 17.9 vs GDN 16.0; short LM Harness 55.1 vs Transformer 55.3.
Mamba, Mamba 2, and Gated DeltaNet pack the whole history into a bounded recurrent state, so compute scales linearly with length. That compression turns the memory law into an architectural choice. These models start from ordinary differential equations, so past tokens fade exponentially and distant events drop out of view. On long sequences, early facts often should still matter. Exponential forgetting is the wrong prior.
Fractional differential equations are the physics tool for hereditary effects: the present depends on a weighted stretch of the past, with an algebraic tail. They are non-Markovian, so the current state needs the entire trajectory and will not fit a finite recurrent layer. FRAC's job is to fold power-law long memory into a selective SSM that still trains with a parallel scan, prefills, and decodes from a bounded state.
Swap the ordinary derivative in the linear relaxation equation for a Caputo fractional derivative. Homogeneous decay becomes Mittag-Leffler, with a long-time tail of order t to the minus alpha. Alpha near 1 behaves like an ODE; smaller alpha forgets more slowly.
The kernel is a continuous mixture of ordinary exponentials. FRAC puts geometrically spaced timescales on a log grid, fits the heavy tail with a finite sum of exponential modes, then discretizes each mode under zero-order hold. Each mode is a scalar first-order recurrence whose retention depends on step size and the effective timescale. Main runs use 16 modes, with base scales from 1 to 2 to the 17, so the slow mode still lives past 64K extrapolation. Think of a dozen half-lives stacked to fake a slower tail.
Selectivity rides on that geometry:
The layer stays a diagonal affine recurrence, trained with a chunked parallel scan. The kernel sets the injection coefficient to 1 minus rho, not the textbook zero-order-hold form (1-rho)/lambda, so timescale and write amplitude can move separately and training stays stable. That is a deliberate break from the derivation.
A single layer of about 200K parameters, trained only at length 512, is tested out to 128K on a heavy-tail probe. FRAC hits 63.8% accuracy at 128K, against 58.4 for GDN, 55.6 for Mamba 3, and 50.0 (chance) for attention. On MADLab compression, fuzzy in-context recall, memorization, and selective copy, FRAC averages 75.4 versus 74.8 for Mamba 3.
1.3B models trained from scratch on 100B FineWeb-Edu tokens at 4K packed length:
| Method | LongBench avg | LM Harness avg | Recall-Retrieval avg |
| Transformer | 14.7 | 55.3 | 38.5 |
| Mamba 2 | 12.0 | 53.8 | 29.9 |
| GDN | 16.0 | 54.2 | 30.9 |
| Mamba 3-SISO | 14.4 | 54.6 | 35.1 |
| Mamba 3-MIMO | 11.4 | 55.3 | 36.8 |
| FRAC | 17.9 | 55.1 | 36.4 |
FRAC is best on 8 of 14 LongBench tasks and 1.9 above the strongest linear baseline, GDN. Past the 4K training length on NIAH, FRAC drops slowest, except on S-NIAH-1, where GDN's gated delta rule filters repetitive filler better. Short-context LM Harness is 0.2 behind Transformer. Recall-Retrieval is truncated to 2K as in prior work: FRAC at 36.4 trails Transformer by 2.1 and sits 0.4 behind Mamba 3-MIMO.
On LCC code completion FRAC scores 11.0 against Transformer's 26.0. That gap is large.
390M ablations, 30B tokens, 16K document perplexity: full FRAC is 47.2 / 28.4 / 11.3 on ProofPile / PG19 / GovReport. Pinning alpha to 1 raises those to 61.3 / 36.2 / 19.7. Dropping the write prior yields 68.5 / 41.6 / 18.2. A pure multiscale bank with all fractional priors removed hits 76.3 on ProofPile. M=8 is worse across the board; M=32 is roughly tied at 5% more parameters and 7% slower compute. Removing feedthrough D sends GovReport from 11.3 to 23.6.
Prefill at 16K is 77490 tokens/s for FRAC, above 74314 for Mamba 3-SISO and 56297 for Transformer. Training throughput is 241598 tokens/s, below Mamba 2 at 278620. The decode state is 65536 scalars per layer, 8 times smaller than Mamba 2/3 and about 4.5 times smaller than GDN.
Changing the memory law is a different axis from adding gates or delta-rule updates on top of exponential decay. FRAC shows a power-law prior can sit in the existing scan pipeline without collapsing short-context scores, and it buys a modest, consistent lift on long extrapolation and LongBench. The state is an order of magnitude smaller than Mamba's while remembering farther, because the 16 modes integrate over timescales, not by growing the recurrent tensor.
Anyone already on Mamba or GDN can swap the mixer. Precise associative retrieval is still open: GDN wins S-NIAH-1, and the paper's own next step is to combine this transition with a delta rule. Short context is essentially tied. This is a long-context incremental change, not a reason to drop Transformer.
The implementation leaks away from the theory. The injection coefficient was changed. The write prior is not uniquely fixed by the fractional kernel; it copies the read functional form. A sum of exponentials on a bounded horizon is not fractional dynamics on an infinite line.
Training is always 4K packed. 64K NIAH is zero-shot extrapolation. Recall-Retrieval at 2K is short retrieval. Eight of 14 LongBench tasks average under 8K tokens, so the headline mean is diluted by short tasks. LCC falling to 11.0 is unexplained. The DNA experiment is a figure with no per-length numbers in the text. Ablations stop at 390M / 30B tokens; the 1.3B model is not dissected the same way. There is no hybrid-attention comparison and no head-to-head with Mixture-of-Memories style capacity routing.