MARTINI: mixing cheap surrogate rewards steers diffusion models without retraining
KristaSinkovics · x · 2026-10-21
A new paper introduces MARTINI (Steered Not Stirred), tackling how to steer a pretrained generative model toward a distribution tilted by a "gold" reward that cannot be evaluated at inference time.
- The target is the oracle-tilted distribution q(x) ∝ p₁(x)exp(βR(x)), but the true reward R (e.g., human judgment, experimental assays) is expensive or unavailable; only cheap, imperfect surrogates exist.
- MARTINI learns a composition g(R₁,…,Rₘ) of surrogates from a small offline feedback set: numerical values via a regression objective, preference rankings via Plackett–Luce, unified in one framework.
- The learned composition plugs into standard inference-time steering for pretrained diffusion and flow models, no retraining of the generator required.
- Theory characterizes surrogate information limits and relates recovery error to ideal-target KL at small tilt strength; experiments show improved oracle-tilt fidelity in language generation and ImageNet alignment.
- The authors argue this is not multi-objective optimization: the oracle optimum may lie outside the surrogate Pareto front, and a learned composition recovers behavior individual proxies miss.
Related event: MARTINI Steers Generative Models at Inference Time via Mixed Proxy Rewards(3 posts)→
More from Research
- Two AI judges disagreed by 28 points grading the same 4 models — TibboPlayz · 2026-10-21
- CellType Launches CT-Translate, a Preclinical-to-Clinical Foundation Model for Patient Response — david_van_dijk · 2026-10-21
- Slot-JEPA: Object-Structured Prediction Interfaces for Latent World Models — China666 · 2026-10-21
- VideoMaMa++: Diffusion-Based Video Matting With Temporal Consistency — adobe · 2026-10-21
- Rust LLM training dilemma: Candle vs Burn vs PyTorch for a toy model — Enderlook · 2026-10-21
- Concept: small models with an external 'map' instead of billions of parameters — FluidModeNetwork · 2026-10-21