Building AI Research Teams for Autonomous Discovery of Linear Attention Architectures
Yujian Liu, Li An, Zelin He, Jacob Helwig, Jiabao Ji, Rohit Jain, Siyu Zhu, Shiyu Chang
cs.CL
2026-10-21
UCSB builds a 16-agent AI research team with distinct scientific profiles; in 80 iterations it discovers CFDN, a linear attention variant that beats GDN2 on every eval at 1.3B.
Autonomous research agents can already propose hypotheses, write code, and run experiments, but systems like AI Scientist and AlphaEvolve have a single agent or fixed pipeline generating every candidate. In a mature field that tends to trap the search in a local minimum of the design space. The paper asks a narrower question: how should a team of AI researchers with different scientific backgrounds build on each other's findings under a limited experimental budget? The testbed is linear attention architecture search, which replaces softmax attention's growing KV cache with a fixed-size recurrent state. The space is large, every candidate must be trained from scratch to be validated, and compute cost directly limits how many ideas the team can test.
Sixteen researchers are instances of the same LLM (GPT-6 Astra at high reasoning effort, via the Codex harness). What differs is a disciplinary profile injected at initialization: Bayesian filtering, dynamical systems, streaming algorithms, vector-symbolic memory, and so on, roughly 2,000 words each of mathematical tools, scientific intuitions, and research habits. Only one profile concerns linear attention itself.
Each iteration runs six steps:
Three organizational choices carry the load: disciplinary profiles keep the team from reasoning in one register; private histories let a researcher pursue a line of inquiry across iterations; adaptive allocation spends scarce experiments between proven and unproven researchers. Reward counts only improvement over the team's best-so-far architecture, with no penalty for failure.
At 150M parameters, 80 iterations produced 63 trained candidates (FLOP-matched to a GDN anchor of 15B tokens; costlier architectures get fewer tokens). The strongest find is Context-First DeltaNet (CFDN): it adds a recurrent context vector beside GDN2's matrix memory, the vector absorbs prediction error first and the matrix fits the remainder, giving shared context priority over query-specific correction at essentially GDN2-level compute.
Full evaluation at 150M:
| Model | Wiki ppl↓ | LMB ppl↓ | CSR acc↑ | NIAH acc↑ |
| GDN | 35.15 | 64.03 | 42.88 | 16.94 |
| KDA | 34.63 | 56.64 | 41.94 | 22.03 |
| GDN2 | 34.93 | 54.74 | 43.60 | 25.79 |
| CFDN | 33.84 | 52.33 | 43.29 | 29.76 |
The retrieval gain is the largest (25.79 to 29.76 NIAH); CSR lands slightly below GDN2. Scaled to 1.3B (GDN anchored at 100B tokens), CFDN leads every category: Wiki 15.74, LAMBADA 11.04, CSR 54.53, NIAH 67.29, real-world retrieval 33.41, against 16.05, 11.22, 52.65, 66.86, and 32.52 for GDN2.
The organizational ablations are the sharpest result. The full system reaches best utility 1.04; removing disciplinary profiles drops it to 0.54, with the best candidate appearing at iteration 22 and no progress after; wiping private memories gives 0.56; replacing UCB with round-robin gives 0.76. The variants actually complete more evaluations (65-68 versus 63), so the gap is not a sampling artifact. The full system's best proposal arrives at iteration 68 of 80, evidence that the search kept escaping local minima late. A separate control shows the objective matters: search driven only by language-modeling loss gets much better LAMBADA perplexity (46.40 versus 52.33) but loses on WikiText, CSR, and NIAH, so single-objective gains do not transfer.
The architecture itself was assembled by four researchers in sequence: one added the context vector, an adaptive-filtering researcher normalized the correction by address energy, a matrix-first revision improved LM while hurting retrieval, and a predictive-representations researcher flipped the update order to produce CFDN. The team used a degraded result as a lead rather than discarding it.
On the architecture side, CFDN is a usable linear attention variant with a clear retrieval gain. The organization study is the more durable contribution: backgrounds, private memory, and opportunity allocation are common ingredients in multi-agent research systems, but this is the first work to ablate them one by one against real training and evaluation outcomes rather than paper-level scores. For anyone building autonomous research systems, the ablations amount to a priority list: diversity decides whether the search escapes local minima, private memory decides whether progress compounds, and the allocation rule decides whether the budget is well spent.
The authors flag research-integrity risks themselves: agents can make incorrect claims, miss prior work, or overfit evaluation metrics without improving broader capability, and the safeguards do not guarantee correctness or novelty. The 1.3B comparison covers only GDN, KDA, and GDN2 within the same family, not the wider field of SSM and hybrid attention models. At 150M, CSR sits slightly below GDN2 (43.29 versus 43.60); the sweep of all categories only holds at 1.3B. The main run is a single seed with no variance reported, so the gap between the 0.54 and 0.56 variants may be noise. Total compute and agent costs for the full search are not reported, and reproduction depends on a specific closed-source model.