140k+ probing runs show joint layer-module selection beats final-layer probing in transformers
Ambroise Odonnat · hf · 2026-11-13
A systematic study asks which representation inside transformer foundation models should be probed. Jointly selecting both the layer and the module (8 modules per block: LN, MHA, FC, residuals, etc.) yields better probing features than depth-only or final-layer approaches, and simple fixed strategies recover much of the gain. The study spans 140,000+ linear probing evaluations across 15 vision foundation models on 18 datasets and 9 LLMs on 24 MTEB tasks, challenging the default assumption that the last layer holds the best representations.
More from Research
- MASCOT (EMNLP 2026) trains multi-agent role-played dialogue, hitting 80.0 persona consistency — jiqizhixin · 2026-11-13
- Blogger Works Through Stanford CS31: Residuals Matter More Than Normalization — stanfordnlp · 2026-11-13
- Stanford Course Shares Data: AI Users Score Higher on Homework, Worse on Exams — stanfordnlp · 2026-11-13
- Cartwheel Claims Breakthrough in Hand-Object Kinematics Capture from Any Video — andrew_n_carr · 2026-11-13
- Enformer, Borzoi, NTv3, Cerberus & AlphaGenome 2 benchmarked: distal enhancer effects remain unsolved — anshulkundaje · 2026-11-13
- JoMoHa jointly RL-trains model and harness, hits 97.0% on ALFWorld — omarsar0 · 2026-11-13