Princeton/Meta evolve inference strategies: 39% cheaper Qwen3 runs, same accuracy
michaelTM_ai · reddit · 2026-11-23
Summary of arXiv 2611.03618 (Princeton + Meta): a strategist agent (GPT-6 Sol via Codex CLI) writes inference strategies as code — chains, verifiers, voting, routing across Qwen3 0.6B/1.7B/4B with thinking on/off — tests them on real samples, and stores each with measured accuracy and cost in a shared library that future tasks start from.
Results: across 30 benchmarks, matches the best hand-built baseline (74.7% vs 74.0%) at 39% lower inference cost, winning 24; routing is credited — each step picks its own model size and thinking mode.
Transfer: strategies found on Qwen3 run unchanged on GPT-5 and Gemini 2.5, still beating baselines; on 10 unseen benchmarks, library retrieval alone gets within 2 points everywhere and wins 8.
Library > loop: same search with the library off drops to 65.4% vs 78.7%.
Winner's curse: chosen strategy is +4.3 on the selection split but only +0.7 on held-out data — expect pipeline numbers tuned on your own eval to shrink similarly.
Caveats: small dev splits (≤100 samples), loses/ties on 6 of 30 (incl. AIME, MedQA), some wins are costly (BoolQ +16% at 31x cost), and search cost isn't in the 39%.
More from Models
- Tesla FSD V15 renders real-time 3D street view at night — blurry store signs reveal it's AI-generated — lxfater · 2026-11-23
- Distillation is mostly cope: labs buy astronomical data, dev argues overhyped in local AI community — gnukeith · 2026-11-23
- Sarvam's new 105B checkpoint jumps from 42.6 to 73.9 on benchmarks, claims Flash-Lite Pareto frontier — itsOmSarraf_ · 2026-11-22
- Dev says Google models are making real progress on lossless compression as intelligence proxy — gnukeith · 2026-11-22
- Astra-6.2 'lazy' backlash blamed on user education: default manual mode vs goal mode — alexisgallagher · 2026-11-22
- Musk: Tesla has had a near pixel-perfect world model since last year — giffmana · 2026-11-22