Generalizing 'Next Token' to 'Next Joint' Prediction: Tensor Networks Break Action Space Bottlenecks
kalomaze · x · 2026-08-12
Developer @kalomaze proposes generalizing the standard 'next token prediction' mechanism in LLMs to 'next joint prediction'.
- Core Insight: Without relying on diffusion, MSE regression, or flow matching, conditioning specific parameterizations on a sufficiently rich Transformer hidden state can optimize for exactly valid joints over combinatorially massive spaces.
- Potential: This opens a path for exact likelihood and policy gradients over far larger action spaces than currently expected, bounded primarily by a rank bottleneck.
- Theoretical Basis: The discussion references a 2017 paper on unsupervised generative modeling using Matrix Product States (MPS, a tensor network from quantum physics), highlighting its capabilities in machine learning generative tasks.
Related event: Tensor Train Decomposition Enables Exact High-Dimensional Modeling(4 posts)→
More from Research
- The Bitter Lesson of AI Agents: Why Simple Grep Beats Complex RAG — paraschopra · 2026-08-12
- Architectural Deep Dive: What Actually Changed Between Meta's Llama 3 and Muse — Hesamation · 2026-08-12
- Fields Medalist Timothy Gowers Asks: What Sort of Maths Are LLMs Good At? — stevenstrogatz · 2026-08-12
- Falcon-Perception 0.6B Enables Efficient Local Image Segmentation — vanstriendaniel · 2026-08-12
- CEPR Paper: Framework for Measuring Firms' AI Efforts and Economic Impact — TaniaBabina · 2026-08-12
- LeAD-M3D: LiDAR-Free Monocular 3D Detection Achieves SOTA at 3.6x Speedup — rsasaki0109 · 2026-08-12