KDD paper: naive 8B LLM critic drags AutoML search below random search, up to 0.245 AUC lost

rohanpaul_ai · x · 2026-11-18

An Amazon + UMass paper accepted at KDD 2027, "The Critic Can Hurt," systematically audits LLM critics in agentic AutoML search loops.

Key findings:

The authors will release CRITFAULT, the first benchmark for detecting harmful critic feedback in AutoML. Takeaway: critics aren't free — audit before trusting and gate when capability or verification is insufficient.

Original post →

More from coding & agent

coding & agent channel →