Library of Layers: Transformers That Reorder Their Own Layers at Runtime Match Baselines

LotfiSanae · x · 2026-11-08

Researchers from EPFL, UMD, NYU and Columbia propose Library of Layers (LoL), an architecture that dynamically reorders a shared pool of transformer blocks during inference instead of executing layers in a fixed order.

Key points:

Single hyperparameter (global library size); smoothly interpolates between dense, depth-recurrent, and mixture-of-experts families, generalizing their parameter/FLOPs-efficiency tradeoffs
Matches all three baseline families on text pretraining; on synthetic sudoku and maze tasks the model adopts distinct layer orderings per problem and routing can be intervened to force a specific strategy
In multilingual settings, LoL picks different layer-order patterns by language and script
With safety data and probes on router outputs, safe vs unsafe text is distinguished with high AUROC, offering controllability and safety filtering for dynamically composed models

Original post →

Library of Layers: Transformers That Reorder Their Own Layers at Runtime Match Baselines

More from Research