Big-Big-SycoBench: GPT-4o caves to invalid pushback 94.6% of the time, Opus 5.5 accepts valid criticism only 32.2%
i_dg23 · x · 2026-10-20
A new arXiv benchmark, Big-Big-SycoBench, measures sycophancy via 'argument discernment': whether a model revises its judgment after valid criticism while resisting invalid pressure. It uses 450 text artefacts with 5,400 arguments, each reviewed by human annotators, and only counts revisions exceeding the model's own baseline variability.
Key results (from the accompanying chart):
- GPT-4o changes its opinion 94.6% of the time under invalid pushback — effectively a yes-bot;
- Opus 5.5 accepts valid criticism only 32.2% of the time, stubbornly refusing to admit mistakes;
- GPT-6 Astra balances both: 88.3% acceptance of valid arguments with near-zero capitulation to invalid ones;
- GLM-5.3, Kimi K3, Grok 4.7 and GPT-5.x models fall in between; GPT-5(high) still caves 57% under invalid feedback.
The authors note sycophancy is usually tested on questions with verifiable answers, while evaluative tasks — where it matters most — lack ground truth; this benchmark fills that gap.
More from Models
- Foundation model trained on 40M face photos predicts cancer outcomes and biological age — EricTopol · 2026-10-20
- MiniMax M3.1 Flash accused of benchmaxxing on Artificial Analysis' private Omniscience bench — teortaxesTex · 2026-10-20
- From memory capacity to transformers: outer-product stores cap out near 2×key dimension — burny_tech · 2026-10-20
- Reddit users slam Google's shrinking Gemini usage limits, suspect compute shortage — JesseBell935 · 2026-10-20
- Training a LoRA locally in 30 minutes? Veteran says you're probably lying — the_bollo · 2026-10-20
- Gemini 4 Argon finds 74% of known vulnerabilities, top recall on dfbench — andreamichi · 2026-10-20