DeclineCraft benchmark: iterative RAG lowers models' ability to admit ignorance
brunofmr · x · 2026-11-11
The DeclineCraft benchmark (arXiv 2610.28340) uses post-cutoff Minecraft features to build unanswerable questions, evaluating 27 LLMs on Correct Abstention Rate. Single-shot retrieval raises CAR in 38/40 configs, but iterative RAG significantly lowers it in 30/40 (mean 46.4%) with little accuracy change; enabling thinking without retrieval also lowers CAR — suggesting models treat long reasoning chains as a proxy for having found the answer.
More from Research
- Yann LeCun: 'If you are interested in human-level AI, don't work on LLMs' — StasBekman · 2026-11-11
- FlashAttention co-creator Dan Fu on kernels, inference economics, and owning vs leasing GPUs — realDanFu · 2026-11-11
- OpenTac debuts at CoRL as researchers call tactile the missing half of robot data — anand_bhattad · 2026-11-11
- Geodesiclab scores big improvement on antibody structure prediction, the hardest part of the problem — QuanquanGu · 2026-11-11
- AI drug discovery hits its HBM moment as the bottleneck shifts to the wet lab — QuanquanGu · 2026-11-11
- 26,040 agent runs across 21 harnesses and 5 open models: does switching models mean switching harnesses? — Double-Entertainer62 · 2026-11-11