144,000 probing evals show Act@100 and LN2@60 beat last-layer probing in transformers
CatAstro_Piyush · x · 2026-11-17
A new paper from Noah's Ark Lab, EPFL and collaborators runs 144,000+ linear probing evaluations across vision foundation models and LLMs to answer which transformer representation to probe.
Key findings:
- Layer and module choices must be made jointly: the best layer-module pair beats the final-layer residual stream far more than the best layer with the residual stream fixed
- Module performance varies substantially across layers—useful representations can't be characterized by depth alone
- Two fixed, search-free rules recover much of the gain: final-layer feedforward activation (Act@100) for vision, second normalization layer at 60% depth (LN2@60) for language, recovering 42% and 86% of best-pair gains on average
- Vision gains grow with pretraining-to-downstream transfer difficulty
Paper, code and HuggingFace resources are public.
More from Research
- NVIDIA open-sources Nemotron-Labs-Sigma: continuous diffusion LLMs in 14 checkpoints at 3B/8B — ArashVahdat · 2026-11-17
- USC paper proposes a Zeta Law for predicting when biomedical data is enough for discovery — PTenigma · 2026-11-17
- Layer-hot decoding experiment on M4 fails: negative-result spec decoding writeup — azterizm · 2026-11-17
- ByteDance-backed team open-sources Axion1.0: two-stage SFT+RL agent training beats GPT-6 on HLE — 机器之心 · 2026-11-17
- Scientific American editor builds open-source Deep Time Explorer with Claude — anselm · 2026-11-17
- E11 Bio hiring ML scientist to automate proofreading of petabyte-scale brain connectomics data — anderssandberg · 2026-11-17