Rachel Thomas threads papers on LLM failure to follow instructions, starting with Inverse IFEval
math_rachel · x · 2026-10-19
fast.ai co-founder Rachel Thomas threads papers and benchmarks on LLMs' failure to follow instructions. The first is Inverse IFEval (arXiv:2509.04292), a benchmark measuring models' ability to override SFT-induced cognitive inertia and comply with counter-intuitive instructions. It spans 8 challenge types (question correction, intentional textual flaws, code without comments, counterfactual answering, etc.), with 1012 curated Chinese/English questions across 23 domains under an optimized LLM-as-a-Judge framework. Later entries in the thread cover reasoning rigidity and long-context preference following.
Related event: Rachel Thomas highlights research on LLM instruction-following failures(3 posts)→
More from Research
- RFD4-Proteina's 'underrated' docs detail partial diffusion for protein redesign — AllThingsApx · 2026-10-23
- MPI-Sec PhD Thesis on Vulnerability Detection Benchmarks Includes Pre-LLM Classic "Top Score on the Wrong Exam" — mboehme_ · 2026-10-23
- LiveNerf's 29-day data finds no nerfing of Claude Opus 5.5 — TheOnlyVibemaster · 2026-10-23
- New paper makes 9 falsifiable predictions toward a major AI-enabled terror attack by 2027-2028 — StephenLCasper · 2026-10-23
- HumanoidPose recovers humanoid robots' full-body joint states from RGB video alone — cholz · 2026-10-23
- Sophont's Holon-S1 predicts genes and biomarkers from routine pathology slides — iScienceLuvr · 2026-10-23