Princeton/Meta evolve inference strategies: 39% cheaper Qwen3 runs, same accuracy

michaelTM_ai · reddit · 2026-11-23

Summary of arXiv 2611.03618 (Princeton + Meta): a strategist agent (GPT-6 Sol via Codex CLI) writes inference strategies as code — chains, verifiers, voting, routing across Qwen3 0.6B/1.7B/4B with thinking on/off — tests them on real samples, and stores each with measured accuracy and cost in a shared library that future tasks start from.

Results: across 30 benchmarks, matches the best hand-built baseline (74.7% vs 74.0%) at 39% lower inference cost, winning 24; routing is credited — each step picks its own model size and thinking mode.

Transfer: strategies found on Qwen3 run unchanged on GPT-5 and Gemini 2.5, still beating baselines; on 10 unseen benchmarks, library retrieval alone gets within 2 points everywhere and wins 8.

Library > loop: same search with the library off drops to 65.4% vs 78.7%.

Winner's curse: chosen strategy is +4.3 on the selection split but only +0.7 on held-out data — expect pipeline numbers tuned on your own eval to shrink similarly.

Caveats: small dev splits (≤100 samples), loses/ties on 6 of 30 (incl. AIME, MedQA), some wins are costly (BoolQ +16% at 31x cost), and search cost isn't in the 39%.

Original post →

More from Models

Models channel →