Latent feedback cuts pretraining compute by 46% in 4 lines of code
_arohan_ · x · 2026-11-14
New research introduces Latent Feedback: feeding a Transformer's own top-layer hidden states back into the input, so every token and FLOP counts for more.
Key results:
- Same data: 15% less training compute at 100 unique tokens/parameter, 46% less at 400 (2-epoch budget)
- Same epochs: 10% less unique data at 60M params, 20% less at 778M (1 epoch)
- Downstream (109 tasks in 37 suites): ahead in all 17 settings at a 4-epoch budget
- A fitted scaling law predicts savings keep growing with model size
The authors argue high-quality data is the bottleneck of modern pretraining, and the method needs only 4 lines of code.
More from Infra
- North Dakota town votes 3-to-1 against data center restrictions, expects $1.65M in taxes — kevinnbass · 2026-11-14
- InferenceForge claims 43% higher LLM inference throughput than best AgentX baseline — kaiwei_chang · 2026-11-14
- Redditor benchmarks a DGX Station: GB300 throughput numbers revealed — CookEasy · 2026-11-14
- Hugging Face launches llama.cpp v1 engine to run AI models locally everywhere — LinkedInNews · 2026-11-14
- China Took 59% of 600,000+ Industrial Robots Installed in 2025; Humanoid Shipments Still Tiny — ingliguori · 2026-11-14
- SGLang joins NVIDIA AI Enterprise with NIM-certified inference microservices — ying11231 · 2026-11-14