ArchitectureIQ: On the Measure of Training Intuition
Zirui Ren, Shaoyang Guo, Chencheng Tang, Jinxin Wang, Chengyu Xiong, Shanbin Yu, Peihang Li, Yidi Wu, Bangzhe Huang, Qingyu Qu, Leqian Yang, Ziming Liu
cs.AI
2026-09-30
Frontier models pick the winning recipe on ~76% of 500 synthetic 3-choice questions (chance 33%). Best of 10 researchers: 66%. Architecture-only: human 65%, GPT-6 Astra 38%.
Someone who trains models for a living will often kill a bad configuration before the run starts. Language-model agents already edit code, launch jobs, and revise from logs. MLAgentBench, MLE-bench, and the AI Scientist score that whole loop. ArchitectureIQ scores only the first step: one dataset, several training recipes, no execution, pick the recipe with the best test metric.
Public datasets are a leak. A frontier model may already have seen the answer in pretraining. Every dataset here is synthesized for the question, and both the generator and the training program are in the prompt. In principle the prompt fixes the outcome.
The pipeline does not branch. A dataset spec is rendered to code and executed, so the train and test sets are fixed. Recipes are then drawn from controlled pools of models, optimizers, losses, batch sizes, and budgets. Each recipe is a runnable program. Ground truth is the recipe with the best metric across 10 random seeds, so one noisy launch cannot write the label.
The static set is 500 three-choice questions, chance 33.3%, split evenly. Optimizer-only items lock the architecture and vary learning rate, momentum, adaptive methods, and weight decay. Architecture-only items lock the optimizer and vary width, depth, normalization, residuals, and activations. Mixed items leave both free.
Figure 2 shows the grain of a question. The target is y = x^2 + cos(4x). Three recipes share a 4x128 ReLU MLP with 66,945 parameters and differ only in optimizer, learning rate, and weight decay. RMSprop finishes far below the two Adagrad options. The comparison that matters is effective step size: how far the weights actually move in a short budget once learning rate, momentum, and the adaptive denominator are combined.
A second loop asks whether those judgments compress. Claude Opus 5 solves fresh questions with at most 20 propositions in the prompt. It may cite an old one or write a new one, and it splits the credit on that question so the shares sum to 1. An epoch is 50 new questions in parallel. Execution pays the reward: cited propositions gain weighted success if the pick is right, and weighted failure if it is wrong. Between epochs a PUCT score chooses what to expose next, trading current credibility against how little a claim has been tested. PUCT is the explore-versus-exploit score from tree search. A separate model merges duplicates and turns broad rules into conditional sentences. The loop runs 8 epochs.
Human scores are on a 50-question subset. Model scores on the leaderboard are on all 500.
| Method | Setting | Accuracy |
| GPT-5.6 Sol | 500 questions | 76.4% |
| Claude Opus 5 | 500 questions | 76.0% |
| Best human | 50-question subset | 66.0% |
| Mean of 10 researchers | 50-question subset | 51.0% |
| Chance | 3 choices | 33.3% |
| Leading models | Optimizer only | 85–87% |
| Leading models | Architecture only | 58–61% |
| Best human | Architecture only | 65% |
| GPT-6 Astra | Architecture only | 38% |
| In-domain logistic regression | 87 spec features | 74.2% |
| Tree, depth at most 4 | Same features | 61.8% |
| Effective-learning-rate policy | Grouped out-of-fold | 78.2% |
| GPT-4o, empty base | 50 questions | 53.2% |
| GPT-4o, snapshot 8 | At most 20 items | 67.6% |
Paired items keep the recipes and change only the dataset. When the winner stays the same, accuracy is 81–95%. When the dataset flip changes the winner, accuracy drops to 46–50%. Models switch their pick on 6–23% of pairs.
Ten demonstrations do not help in any stable way. On a subsample, Claude Opus 5 is 70.0% both zero-shot and 10-shot. GPT-5.6 Luna goes from 64.0% to 60.0%, DeepSeek V4 Flash from 62.0% to 65.0%, DeepSeek V4 Pro from 62.0% to 60.0%. Humans working in order do not improve as they go. On a fixed 50 questions, sweeping reasoning effort from none to max stays inside sampling noise: gaps under about 7 points cannot be told apart. The only gain called out, for GLM-5.2, sits in the first few hundred output tokens.
With 2, 3, 5, and 10 choices, four models follow acc(k) ≈ a + (1-a)/k, R^2 at least 0.94, with fitted a between 0.35 and 0.39. The paper treats a as the share of questions the model actually knows. The rest of the formula is guessing. Among the cited entries in Figure 6, optimizer and data/loss items land around 74–86%, architecture items around 63–65%. Claude Opus 5 with the base peaks at 76% on snapshot 4, then sits near its original level.
An auto-research agent spends its budget on the run. This benchmark prices the call made before that spend. The 76% headline is mostly short-horizon optimizer habit: a learning rate that is too small never leaves initialization, and an adaptive method gets off the chance-loss plateau sooner.
Twenty readable conditionals, with no weight update, move GPT-4o from 53.2% to 67.6%. The strongest solver barely moves. The base reads as pretrained experience written down.
A development-stage analytical rule, 78.2% out of fold, also clears the best zero-shot model. On these short synthetic tasks the language-model score is not a residue that rules cannot reach. The open gap is how architecture meets the data. Longer chain-of-thought did not close it.
The paper states its own scope. Data are synthetic, runs are short, and the architecture family is a finite registry. That is not the scale or the data of current large-model training. Transfer to a more realistic generator is untested.
The human 66.0% is 50 questions, not 500. The same paper says that on 50 questions, gaps under about 7 points are not distinguishable. A 66-versus-76 gap supports a direction. It does not show a stable win over top researchers. The mean of the ten is 51.0%. The abstract asks about top researchers; the sample is the best of ten machine-learning researchers.
On architecture-only items the order flips. Best human 65%, leading models 58–61%, GPT-6 Astra 38%. Human intuition beats the models on that slice.
The abstract says about 20 knowledge items let GPT-4o almost match Claude Opus 5. The same 50-question plot shows 67.6% against a Claude peak of 76%. The lift is real. The tie is not.
Section 4 uses the in-domain logistic regression at 74.2% to argue that compact heuristics still trail language models. Section 5 then reports a development-stage analytical policy at 78.2% out of fold, above GPT-5.6 Sol at 76.4%. Structured rules are not left behind.
When a dataset change should reverse the winner, accuracy is still 46–50%, above the 33.3% chance line. The rate at which models switch answers runs from 6% to 23%, and the main text does not break it out by model. Insensitivity is what was measured. The claim that models do not understand data steps past that number.