A Hybrid Nested Harness for Decoupling Structure and Parameters in LLM-Driven Optimization
Víctor Gallego
COLM 2026
cs.LG, cs.NE
2026-08-08
LLMs propose structure well but tune parameters poorly. Splitting optimization into an outer LLM sketch and an inner numerical optimizer (CMA-ES, VI, or NUTS) beats pure LLM search and pure numerical optimization across three domains.
In LLM-driven evolutionary optimization (FunSearch, AlphaEvolve, autoresearch), a frozen LLM does two jobs at once: writing code structure (control flow, helper functions) and setting continuous parameters (thresholds, learning rates). The trouble is that LLMs are good at structure but poor at continuous optimization, wasting tokens on discrete jumps inside a trial-and-error loop, while traditional numerical solvers such as CMA-ES are dramatically more efficient. A single artifact mixing two qualitatively different decisions is the crux.
Hybrid nested search splits optimization into a bilevel loop:
The outer loop has a frozen LLM propose a structural sketch with numeric holes, plus a manifest for each hole giving bounds, type (continuous, integer, log, power of two), and a suggested initial value. The inner loop is a numerical optimizer that tunes the holes to the optimum, maximizing fitness f(tau, theta). The inner solver is pluggable: CMA-ES for black-box, gradient routines for differentiable cases, MCMC (VI or NUTS) when optimizing a distribution. A structure is scored on its tuned value and then promoted or rejected (1+1 rule, or population-based / GEPA).
The key idea is parametric de-aliasing. Vanilla LLM search ranks structures by the untuned guess, so a good structure paired with a bad guess is wrongly discarded. Nested search ranks by the tuned optimum, letting the LLM see the structure's real potential, and the advantage is proportional to the tuning gap. The inner-loop cost is negligible relative to LLM calls here.
Three problem families compare three strategies: vanilla LLM joint search, numerical-only on a fixed structure, and the hybrid. Inner budget is 100, and the objective is withheld from the LLM. Models tested: Opus 4.8, GLM-5.2, Gemini 3.5 Flash.
| Family | Representative result | Vanilla | Numerical | Hybrid |
| Meta-optimizers | rastrigin (loss down) | 19.2 | 9.95 | 1.22 |
| Cloud systems | Cloudcast inter-cloud (cost down) | 213 | 317 | 169 |
| Social dilemma | Cleanup heavy (welfare up) | 0.480 | -- | 0.582 |
| Bayesian inference | gaussrot (VI, nats) | -33.4 | -- | +4.30 |
On meta-optimizers, hybrid improves delivered loss by up to twelve orders of magnitude on ackley and rosenbrock and even beats oracle CMA-ES on the multimodal rastrigin (1.22 versus 9.95); the two tie on ellipsoid. On cloud-system policies, inter-cloud Cloudcast routing with hybrid ($169, Gemini) beats vanilla ($213) and pure CMA-ES ($317), a 1.9x improvement over pure numerical search. On the Cleanup sequential social dilemma, hybrid achieves the highest welfare in all 9 model-by-seed runs, with all three models converging to 0.582 under heavy load; a piecewise ramp policy that is worst untuned (welfare -0.10) becomes best tuned (0.582), which vanilla search discards in favor of a sigmoid that peaks at 0.480, leaving about 21% on the table to aliasing. For Bayesian inference the inner solver becomes VI or NUTS: the affine Gaussian gains +37.7 nats under VI, and the funnel gains +3.12 decades under NUTS, exactly the nonlinear case no mass matrix can fix.
It is a clean, general factorization with pluggable loops and a lightweight interface (the manifest), holding across optimizer families and LLM families. It names the failure mode of joint LLM search (parametric aliasing) and fixes it. With inner-loop cost negligible relative to LLM calls here, it is a directly usable improvement for LLM-driven discovery pipelines.
The inner-loop cost is negligible only because each evaluation is cheap here. When each evaluation is a full training run, precisely the AlphaEvolve-style cases, the tens to hundreds of evaluations CMA-ES needs become expensive, which the authors flag. With a stronger text optimizer such as GEPA on the outer loop, much of the gain is recovered without the inner loop, shrinking its marginal value from about 10-20% to about 5-10%. Results are on benchmarks; payoff on real scientific discovery is not demonstrated.