Optuna creator Akiba: when 2 hyperparameters cost $20M, what's left for BO/HPO

Stefania_druga · x · 2026-10-16

Takuya Akiba (@iwiwi), original creator of Optuna at Preferred Networks, previewed his Optuna Meetup #3 talk in two parts:

• Hyperparameter optimization in the LLM era: Cites the arXiv paper "Predictable Scale: Part I, Step Law", which trained 3,700+ LLMs from scratch across 100B tokens using 1M NVIDIA H800 GPU-hours (¥300M by his estimate) to derive the Step Law — optimal learning rate follows a power law in model size N and dataset size D, while optimal batch size depends mainly on D. Searching hyperparameters with it needs only 0.094% of test compute. He asks what BO/HPO practitioners can still contribute when tuning two hyperparameters costs ¥300M.
• Inference-time scaling resembles Optuna: Shows Sakana AI's official AB-MCTS implementation (treequest) has an API structure strikingly similar to Optuna's (objective/generate callbacks plus optimize/step loops).

The event also covers Optuna 5.0's new features and Rustuna.

Original post →

More from Infra

• RL train-sample systems are frontier AI's hardest infra problem — and a trade secret — brianryhuang · 2026-10-16
• Local tuning of GLM-5.3-Flash on 2×GB10 lifts decode from 49 to 58 tok/s — EAccelerate_42 · 2026-10-16
• 3x DGX Spark cluster runs GLM-5.3-Flash locally behind a proxy, 1.75M tokens/day at 119W — Teknium · 2026-10-16
• Amazon Proposes Per-Request Validated Evidence Caching for Production RAG — _reachsumit · 2026-10-16
• LinkedIn's TransX report: sequence transduction for recsys, +6.02% CTR in A/B tests — _reachsumit · 2026-10-16
• llama.cpp PR boosts CPU inference speed with merged AVX2 tiled accumulators — jacek2023 · 2026-10-16

Infra channel →