W&B's ARIA agent runs 200+ autoresearch experiments, nearly beats its best result live
AI Engineer · youtube · 2026-09-26
At AI Engineer, Weights & Biases principal engineer Tim Sweeney demoed ARIA, an AI research agent that launched and monitored a batch of autoresearch experiments (on Karpathy's autoresearch project) on real GPUs and nearly beat the project's best result live.
How ARIA is built:
- Launches/monitors training jobs, summarizes best runs, finds patterns across 200+ experiments, and generates reports and dashboards; also on the W&B iOS app
- Logs 100% of traces to Weave and runs LLM judges on live traffic to catch signals like user frustration
- Tasks written as YAML "unit tests"; a nightly eval suite (73% vs 72% for the latest candidate) drives go/no-go decisions
Production tips: invest in agent-focused observability, treat evals as your new CI, keep humans reviewing traces, and add value via context and tools before over-engineering the harness.
More from AGI Musings
- levelsio: wealth inversely correlates with regulatory burden, citing Switzerland vs Spain — garrytan · 2026-09-26
- Juan Benet on vibe coding: ship first, optimize later — models now do real engineering — juanbenet · 2026-09-26
- Sherry Turkle on chatbots as objects of emotional attachment and the risks of grieftech — PeterBowdenLive · 2026-09-26
- ResolVI models measurement errors to fix single-cell RNA data, cutting false signals from 14.8% to under 0.01% — bravo_abad · 2026-09-26
- Daniel Lemire: AI can now write theses, so what is a PhD worth? — lemire · 2026-09-26
- Unsafe AI is less useful: safety is a prerequisite for faster adoption — mchorowitz · 2026-09-26