Amazon paper: weak LLM critics can sink search loops, ideal gain only 0.005 AUC

rohanpaul_ai · x · 2026-11-18

LLM critics may hurt more than help

A new paper from Amazon and UMass Amherst finds that adding an LLM critic to an AutoML search loop—one that reads past results and suggests what to try next—has tiny upside and much larger downside.

Setup

Key results

Takeaway

Before trusting an LLM critic, even a large one, run the same budget without it and drop it if it loses. Teams usually check whether critics help, rarely whether they hurt.

Original post →

More from coding & agent

coding & agent channel →