Mathematician shares a cheap 4-step heuristic for hyperparameter tuning
dejanseo · x · 2026-09-23
Mathematician Sylvain Deaure shared his usual 'not clever but cheap' heuristic for tuning parameters:
- Coarse seeding: hand-pick a few realistic starting points at very wide spacing to establish bounds, not precision; score by absolute metric, not rank.
- Sensitivity screening: from the best seed(s), move each parameter alone with a large step to get a rough local slope—usually 2-3 parameters do all the work.
- Coordinate search: optimize the most sensitive parameter with the rest frozen.
- Interaction check: hold it at two levels and sweep the second most sensitive; if the optimum shifts, they're coupled and you search that pair jointly.
More from Research
- New paper reframes score distillation as distribution matching, explains SDS mode collapse — burny_tech · 2026-09-23
- Cambridge publishes open-access 458-page book on nonparametric and high-dim stats — FrnkNlsn · 2026-09-23
- Reinforce-Ada: adaptive sampling for RLVR recovers lost signals, 2x faster convergence — burny_tech · 2026-09-23
- Paper: Transformers do have world models — failures trace to feature interference — burny_tech · 2026-09-23
- Paper suggests transformers can't learn loop-closure in world modeling — burny_tech · 2026-09-23
- Epoch AI: The plunging price of thought — intelligence keeps getting radically cheaper — Proper_Actuary2907 · 2026-09-23