Library of Layers: EPFL's architecture reorders transformer layers at inference
Abdellah Rahmani · hf · 2026-11-03
Researchers from EPFL, UMD, NYU, and Columbia propose Library of Layers (LoL), an architecture that dynamically orders its transformer layers at inference to fit the task. With a single hyperparameter—the size of its global layer library—LoL smoothly transitions between parameter reuse and sparsity, unifying dense, depth-recurrent, and mixture-of-experts parameter/FLOPs tradeoffs. It matches all three baselines on text pretraining; on synthetic sudoku and maze tasks it employs distinct layer orderings for different solution strategies, and routing can be intervened to force a specific strategy, with interpretable routing in multilingual and safety settings.
More from Research
- Mapping UK's AI-for-Science landscape: the 'DeepMind mafia' seeds a new wave of startups — ersatzben · 2026-11-03
- Researchers map ~600 UK AI-for-Science organisations as VCs bet on an undefined industry — ersatzben · 2026-11-03
- How streaming speech-to-speech models decide when to speak: READ/WRITE gate architecture — Gauri_the_great · 2026-11-03
- Yes, tequila can grow diamonds: 2008 paper grew diamond films from Tequila — secemp9 · 2026-11-03
- EEG study: autism alters feedback evaluation, not raw perception — MacrinePhD · 2026-11-03
- Free guide by de Chaisemartin's team helps you pick the right DiD estimator — avicgoldfarb · 2026-11-03