Princeton team says OpenAI's new architecture closely resembles its T2MLR paper
prfsanjeevarora · x · 2026-09-05
Princeton professor Sanjeev Arora says OpenAI's new architecture has several close precedents in recent works including his group's T2MLR, whose training code was written over a year ago and submitted to an ICML workshop in March — rejected, and never arXiv-ed in time.
T2MLR (Transformer with Temporal Middle-Layer Recurrence) key points:
- Fuses a cached middle-layer representation from the previous token into an earlier layer of the current token, letting intermediate reasoning states persist across decoding steps with little overhead
- Consistently beats data- and parameter-matched Transformer baselines on pretraining and multi-hop reasoning finetuning
- Recurrence on a localized middle-layer block (as little as 20% of the network) often outperforms full-layer recurrence
- No pretraining from scratch needed: retrofitting the recurrent pathway into a pretrained 1.7B Transformer with brief finetuning substantially improves math reasoning
More from Research
- TAOCP open problems released as a dataset to benchmark frontier models — sytelus · 2026-09-05
- Clinic-in-the-Loop: why clinical trials are the real bottleneck breaking Eroom's Law — anshulkundaje · 2026-09-05
- Contentious preprint claims ensemble-wrapped LLM achieves phenomenal consciousness — PeterBowdenLive · 2026-09-05
- Fine-tuning on 'random' numbers transfers teacher bias — subliminal learning worries — ryanorban · 2026-09-05
- Survey of 668 developers: readers who suspect AI writing will stop reading and block you — IanArawjo · 2026-09-05
- Randomized YaRN: training on short context with sampled positions boosts 128K reasoning — gregd_nlp · 2026-09-05