Neural Thickets Revisited: Frozen-Model Logit Perturbations Match Full-Weight Perturbation Ensembles
kastnerkyle · x · 2026-10-31
A Kyutai / Meta FAIR / UC Berkeley / Google DeepMind paper (arXiv 2610.40649) revisits the Neural Thickets hypothesis that task-specific experts densely populate the weight space around pretrained LLMs.
Key findings:
- Gains reproduce, geometric story doesn't: reported improvements are real but don't require nearby weight-space experts.
- Full-weight perturbations are unnecessary: perturbing a single decoder block — or even just the value projection — recovers most of the gains.
- Logit PSV: keeping the model fully frozen and adding fixed random Gaussian biases to output logits matches or exceeds full-weight perturbation ensembles, a training-free alternative.
- First-order analysis: weight perturbations act mainly as an inefficient way to induce structured logit shifts.
- Classical ensembling explains it all: gains stem from output diversity, validation-based selection, and voting (formalized as Perturb-Select-Vote); on strong base models, simple temperature-tuned majority voting matches full-weight PSV.
- Vision extension: on DINOv3 under domain shift (e.g., Earth observation), logit-only perturbation substantially improves OOD performance while full-weight perturbation adds little.
More from Research
- 4B decision model trained on 117k examples matches Jev — except it can't detect sarcasm — SRSchmidgall · 2026-10-31
- David Baker Lab Preprint: Protein Design Cuts CD8⁺ T Cell Immunogenicity — gottapatchemall · 2026-10-31
- Companies pay $300 per paper to survey ICLR/NeurIPS authors — grads can cash in — Icy-Park4287 · 2026-10-31
- Gary Marcus mocks OpenAI's new algorithm: exponential subroutine below a 10^25-variable threshold — AlexTensor · 2026-10-31
- After NVIDIA's BindClaw, every AI4Bio paper now needs an agentic pipeline demo to prove versatility — rishabh16_ · 2026-10-31
- CS312 course post explains sharpness and the edge of stability in LLM pretraining — tatsu_hashimoto · 2026-10-31