A new paper argues dimensionality reduction can invent structure and hide the real kind
burny_tech · x · 2026-07-25
Dimensionality reduction can hide structure as well as reveal it
The post points to a basic intro video on how to avoid mistakes when using dimensionality reduction, after a quoted note from Michael Timothy Bennett.
The key argument in the quoted paper is that:
- dimensionality reduction can reveal structures that do not exist and miss structures that do exist;
- simplicity is a property of form, while generalization is a property of function;
- experiments found that choosing weak constraints over simple forms improved generalization by 110%–500%;
- complexity itself has no causal influence on generalization and may only appear correlated because of confounding.
The piece argues that depending on how function and form are represented, the apparent simplicity–generalization relationship can reverse or disappear entirely.
Related event: Study: Dimensionality Reduction Can Create and Obscure Data Structures(2 posts)→
More from Research
- Hamel Husain says evals beat vibes after Opus 5 costs 6× more and scores worse — HamelHusain · 2026-07-25
- ByteDance- and Monash-led paper turns task experience into weights for software agents — imjustnewatai · 2026-07-25
- Snorkel AI says agent benchmarks should be rebuilt from production traces — AI Engineer · 2026-07-25
- OPUS selects training data in optimizer space and builds a 30M-token benchmark proxy — VoidAsuka · 2026-07-25
- Statistical physics paper studies optimal MLP learning near interpolation — burny_tech · 2026-07-25
- ICML paper says regularized learning often looks Hebbian, while noise turns it anti-Hebbian — burny_tech · 2026-07-25