Large Discovery Models: Empirically-grounded Model-Based Open-Ended Search
Zhongwei Yu, Yan Song, Xue Yan, Anjie Liu, Xingyu Lu, Yihang Chen, Huichi Zhou, Siyuan Guo, Luoyang Sun, Sihan Chen, Xiangning Yu, Jun Wang
cs.LG
2026-08-16
LDM couples an LLM candidate generator with a continually updated Gaussian-process surrogate and acquisition-guided selection, beating LLM-only reflection and Bayesian optimization across program search, antibody design, and molecular optimization, with 62.4% higher Pareto hypervolume on molecules.
Most interesting scientific objectives are expensive to evaluate: synthesizing a molecule, measuring antibody binding, or running a training job each costs real money and wall-clock time. Inference-time LLM search (tree search, best-of-N, verifiers) excels at math and code precisely because verification there is cheap and repeatable. That premise collapses in science. An LLM's own likelihoods and self-assessments neither track external experimental metrics nor provide calibrated uncertainty, especially for novel candidates outside the training distribution.
Bayesian optimization, the classical alternative, learns a probabilistic surrogate from scarce observations and allocates experiments under uncertainty. But it needs a pre-specified representation of the design space and search operators, which breaks down in open-ended combinatorial spaces like molecules, proteins, and programs. Each approach holds half the puzzle: LLMs propose structured candidates without knowing which are valuable; BO knows what is valuable but cannot propose.
The Large Discovery Model (LDM) closes the loop with three components:
The mathematical core is a KL-regularized variational objective: find a search distribution close to the LLM prior while maximizing expected acquisition value. Its closed-form solution is the LLM distribution exponentially tilted by the acquisition function. The two limits recover known paradigms: zero tilt is raw LLM sampling; a flattened prior recovers classical BO. LDM lives in between. In practice a finite pool approximates the tilted distribution: each round the LLM generates N candidates, acquisition scores reweight or top-b select them, the chosen few get evaluated, and observations update the GP and the context for the next round.
One design distinction matters: high surrogate uncertainty on reachable candidates is exploration; changing the search support itself, bringing in design families the current parameterization cannot express, is discovery. LDM implements discovery by letting the LLM restructure the parameterized space when stagnation is detected. The theory section gives a regret decomposition: total regret = discovery gap (the LLM reservoir never reached high-quality regions) + standard GP-UCB optimization error + the shortfall from sampling a tilted distribution instead of exactly maximizing acquisition.
A slow learning loop runs alongside: high-budget search traces, with their acquisition values, are recorded and used to fine-tune a smaller model with acquisition-reweighted supervision, distilling the search policy into weights. The supervision target is acquisition value rather than reward, teaching the judgment of which next experiment is worth running.
One framework, three domains:
| Task | LDM | Best baseline | Note |
| Training-program search (nanoGPT, val bpb) | 0.9342 | LLM-only 0.9767 | Same H100 5-min budget; reduction 0.0727 vs 0.0301, a 2.4x margin |
| Antibody CDRH3 (5 targets) | -104.1 to -113.1 | AntBO -92 to -113.6 | 200-evaluation budget; on par with the specialized method |
| Molecular multi-objective (KRAS G12D hypervolume) | 27.3 | Pure LLM 16.8 / MOBO 16.8 | +62.4% / +63.1% |
Ablations isolate each part. Raising the test-time budget from 4x4 to 8x8 helps; disabling the discovery mechanism and freezing the feature set converges to a worse plateau; posterior-mean-only selection falls behind uncertainty-aware acquisitions late. The pure LLM research loop, with no surrogate and no acquisition signal, plateaus at 0.956 after 875 experiments; the bottleneck is value, not proposal ability. In the distillation experiments a fine-tuned Qwen3.5-9B matches the Qwen3-Coder-30B reference on the antibody task and retains four of five targets in a transfer test where the model never saw the protein task during training.
For anyone building AI for science systems, this is a reusable architectural template: do not treat the LLM as an omniscient researcher. Treat it as a structured-candidate generator and delegate value judgments to an externally calibrated statistical model. The same acquisition logic served all three domains, with only kernels and feature representations hand-adapted, which suggests the division of labor generalizes. The distillation loop also shows a cost-reduction path: search experience amortizes into smaller models.
Behind the generation-to-reasoning-to-discovery framing sits a concrete claim: the reasoning era scaled test-time compute on cheap verifiers, and the discovery era must scale it on expensive empirical feedback, making value calibration the new bottleneck. The claim has backing from negative results: LLMs show no sensitivity to experimental feedback in BO settings, and point-estimated LLM uncertainty is unreliable. LDM deliberately never asks the LLM to estimate value.
The authors' own list: exact GP inference scales cubically with observation count; test-time batch sampling carries heavy LLM inference cost; kernels and features still need per-domain manual work; domains where the LLM's priors are weak (antibody sequences) cannot beat specialized methods decisively; and the loop uses only in-process observations, leaving external databases and prior literature, the unknown-knowns, untouched.
Additional concerns from a close read. All three case studies evaluate against digital oracles (the Absolut! simulator, AutoDock scoring, a fixed training budget); wet-lab noise and drift remain untested, which the authors acknowledge as future work. The headline multiples such as 2.4x are ratios of absolute reductions against an LLM-only baseline with a small denominator, which reads more dramatic than the underlying numbers. And the reservoir-quality assumption in the regret analysis, a probability mass lower bound near near-optimal candidates, is not experimentally verified for real LLMs.