eNTK theory from BERT era applied to modern LLM pretraining, adopted by frontier labs
prfsanjeevarora · x · 2026-08-01
Sadhika Malladi tweeted that ideas from their BERT-era eNTK paper (arXiv:2210.05643) have been applied to modern LLM pretraining and inspired frontier labs. The paper investigates whether Neural Tangent Kernel (NTK) describes fine-tuning of pre-trained LMs, extends to Adam, and validates on 14 NLP tasks. A quoted tweet notes that linearizing a 150M model with a quadratic tracks true loss for up to 10% of training budget.
More from Research
- Paper: Predictable Causal Architecture Exists Prior to the Origin of Life — drmichaellevin · 2026-08-01
- RLHF Data as the Core of the AI Economic Loop: Quality Data Design Determines Model Survival — herbiebradley · 2026-08-01
- Stanford Scholars Explore Generative and Agentic AI for Drug Discovery in Circulation — james_y_zou · 2026-08-01
- APEX-Accounting Benchmark: 58% Tasks Unsolved, Claude Fable 5 Takes the Lead — EdwardSun0909 · 2026-08-01
- Detecting Text Presence in Images: Best Architectures for Binary Classification — Relative-Pace-2923 · 2026-08-01
- Article Explores Integrating AI Across All Aspects of Radiology Education — PRouzrokh · 2026-08-01