621k paper-added changes analyzed: 26% didn't help by the papers' own numbers
latenticlabs · reddit · 2026-11-17
Latentic Labs read 126,758 ML/CV papers in full (ICML, NeurIPS, ICLR, CVPR, AAAI, JMLR etc., 2016–2026) and extracted every change papers added and measured, totaling 621,064 changes and 1,273,440 findings.
Key results:
- 69% of changes helped, 10% had no clear effect, 16% hurt, 5% mixed — 26% didn't help by the papers' own numbers
- 66% of papers report at least one change that didn't help
- ML venues report null/negative results for 31% of changes vs 22% at vision venues
- Most often unhelpful: higher sampling temperature (52% of 122 papers), larger LoRA rank (46%), larger learning rate (46%); least unhelpful: adding a contrastive loss (4% of 1,317)
The full front page and "usual suspects" table are public at atlas.scime.ai.
More from Research
- Latent feedback cuts Transformer training compute by up to 46% with 4 lines of code — IFM · 2026-11-17
- PaperCritic: RL-trained AI reviewers beat frontier models at judging paper soundness — CarnegieMellonU · 2026-11-17
- LoopFill: training-free speculative decoding speeds up looped Transformers — UIUC-CS · 2026-11-17
- SPILLWAY: deterministic defense stops all templated prompt injections on tool-using agents — saintmachines · 2026-11-17
- SJTU's Calpha folds proteins from a general-purpose LLM's internal residue states — SJTU · 2026-11-17
- AllocFlow: end-to-end flow matching turns raw market data into portfolio allocations — Annan Wang · 2026-11-17