LLMs Rank Optimizers at 85% but Are Data-Blind on Architectures, New Benchmark Shows
ZimingLiu11 · x · 2026-10-24
Parts 1–2 of a thread introduce a new benchmark: each question pairs a synthetic dataset with three training recipes, asking which model, optimizer, and hyperparameters yield the lowest test loss, spanning regression, classification, and next-token prediction. Findings: LLMs rank optimizers well (85–87%) but struggle with architectures (38–61%). When only the dataset changes and the winner flips, most models repeat their pick—at or below the 50% data-blind bound—and more reasoning tokens bring no consistent gain.
More from Research
- Why KV sharing works: 60-layer model keeps 21% of KV cache with lower loss — yoavartzi · 2026-10-24
- Researchers warn Opus 5.5 pushes back with silly analysis, risking false negatives — adam_karvonen · 2026-10-24
- Terence Tao: AI turns beautiful mathematics into 'toxic waste' — TheObserverUK · 2026-10-24
- Matt Niessner's lab retreat: CVPR paper reviews and newly discovered dancing — angelaqdai · 2026-10-24
- Mistral's Timestral tops time-series benchmarks with 209M params, built mostly by agents — hkproj · 2026-10-24
- Claim: task-free continual learning in O(1) memory with zero replay and zero backprop — Horror-Celery4869 · 2026-10-24