Study decodes what a Mamba foundation model learns about limit order books
chaumian · x · 2026-10-23
An arXiv paper probes what autoregressive foundation models actually learn when trained to simulate limit order book message flows.
- Setup: an 8.2M-parameter Mamba-3 state-space model trained autoregressively on tokenized message streams from eight US large-cap equities.
- Key finding: like sequence models building board representations in games, it learns an internal market representation — order-flow imbalance, a key driver of short-horizon price moves, is decodable at R²≈0.59, far above controls.
- The representation generalizes to unseen stocks and holds across traded volume and volatility, suggesting general market dynamics rather than ticker-specific patterns.
- The representation is causally steerable: pushing the state along a feature axis shifts the model's next-message beliefs accordingly.
More from Research
- Embedding arithmetic demo: cartoon − animation − comic = diagram — mayfer · 2026-10-23
- Omnii Genome: Single Generalist Model Outperforms AlphaGenome Across All Benchmarks — JosephJacks_ · 2026-10-23
- Domingos Cites New Result Claiming RL Is 'Basically Useless' vs Supervised Learning — pmddomingos · 2026-10-23
- Word Arithmetic on Embeddings: "Bathroom Minus Toilet Equals Kitchen" Beats SAE Probing — mayfer · 2026-10-23
- Wolfram Opens Rule 30 Prizes to AIs: $30,000 for Three Open Problems — cephaloform · 2026-10-23
- Stanford and Databricks use AI agents to reproduce medical findings and test analytical robustness — ChengleiSi · 2026-10-23