AI4AI-Bench: Can LLM Agents Design Better AI Algorithms? Only 46% Even Try

机器之心 · wechat · 2026-08-22

AI4AI-Bench, released by NaversLab under EinsiaAI, benchmarks whether LLM agents can do real algorithmic design rather than hyperparameter tuning; it hit 1M views on X within 7 hours.
It uses 10 real AI research repos (8 training-related, plus weight averaging and one-shot pruning). Evaluation is two-stage: a 4-hour exploration phase (only source code survives) and up to 12 hours of independent rerun scored by a frozen evaluator, with the original repo as baseline.
Across 29 model/framework/reasoning-effort configs (290 runs), Claude Opus 5 is strongest overall. Of 263 valid submissions, only 122 touched "how the model learns" (loss, supervision, update rules) — these averaged 0.226 vs 0.126 for run-level tweaks.
Raising reasoning effort matters most: submissions reaching the algorithm layer jumped from 8% to 64%, median experiments from 4 to 16, and average score from 0.094 to 0.196 — still only 10% of the way from baseline to theoretical optimum.
Three standout cases: turning pruning into a 3-stage distillation pipeline (perplexity 53.4→13+), building a 500x faster evaluation rig before optimizing weight averaging, and bootstrapping Agentic RL with solver-based imitation learning. Common pattern: build a tool to verify your diagnosis before changing the algorithm.

More from coding & agent