ByteDance Introduces Modular TTT: Modularizing Test-Time Training with Systematic Ablation
ByteDance-Seed · hf · 2026-08-10
Test-time training (TTT) views sequence modeling as an online learning problem, but existing methods are typically hard-coded separately, making it difficult to isolate the role of each component. ByteDance introduced the Modular TTT framework, representing the inner learner as a directed acyclic graph (DAG).
- Modular Design: It exposes the fast-weight network, loss function, learning rate, weight decay, and normalization as explicit design dimensions, automatically composing them into full TTT computations.
- Ablation Findings: Small learning-rate initialization, weight decay, and a single-layer nonlinearity improve performance, while MSE and inner-product losses perform similarly. Deeper fast-weight networks and normalization tend to hurt performance due to excessively large activations.
- Model Performance: Guided by these findings, researchers trained 410M and 1.45B parameter models on 10B tokens, achieving training loss and benchmark performance comparable to Gated DeltaNet.
More from Research
- Theoretical Blind Spot of Discrete Diffusion: Fails to Learn Joint Probability Distributions — kalomaze · 2026-08-10
- NeurIPS 2026 Calls for Papers on Physical Understanding for Embodied AI — shaohua0116 · 2026-08-10
- ReASearch: Single LLM Agent Outperforms Specialized Optimizers Across ML Workflows — _reachsumit · 2026-08-10
- Sakana AI Summarizes 'AI Scientist' Progress in End-to-End Research Automation — SakanaAILabs · 2026-08-10
- NBER Paper Explores AI Agent Economics: Plunging Transaction Costs to Reshape Market Design — danielrock · 2026-08-10
- Meta Introduces SYF: An LLM-based Agentic System for Real-time Conversational Recommendations — _reachsumit · 2026-08-10