Paper: AI Research Preference Models Solve Compute Bottleneck
anirudhg9119 · x · 2026-09-01
Problem: Generating ML ideas takes minutes, but testing them requires hours or days of GPU compute. Progress depends on selecting the right idea to run.
Solution: Introduces AI Research Preference Models (RPMs) to predict promising candidates without running all. Built from frozen LLMs in two variants:
- Inference-only: Reasons over plans, code, and history.
- Agentic: Additionally runs small-scale pilot experiments.
Results: Integrated into AIRA-dojo and evaluated on AIRS-Bench, the variants boost the average normalized score from 0.684 to 0.711 and 0.729 respectively.
More from Research
- No-backprop pretraining speedup of 2.3x claimed, inspired by mech interp essay — CatAstro_Piyush · 2026-10-18
- Training Trick: Fine-Tune Only the Last Layers to Avoid the Moving-Target Problem — Sauers_ · 2026-10-18
- Models Can Be Trained to Sense Activation Probes and Generalize Across GPUs Zero-Shot — Sauers_ · 2026-10-18
- Models Can Learn to Detect Training vs. Deployment and Generalize to New Runtimes — Sauers_ · 2026-10-18
- How One Attention Head Acts as a Switch to Amplify Tiny Probe Differences — Sauers_ · 2026-10-18
- Model trained to express different preferences only when its activations are probed — Sauers_ · 2026-10-18