Inverse IFEval: A New Benchmark Testing Whether LLMs Can Follow Out-of-Distribution Instructions
math_rachel · x · 2026-10-19
Rachel Thomas highlights Inverse IFEval, a benchmark measuring models' ability to follow out-of-distribution instructions, including:
- Inherently flawed questions
- Code without comments
- Not using bulleted lists
- Instructions that change mid-interaction
"The ability to reliably follow out-of-distribution instructions while suppressing rigid training biases is a critical dimension of robustness" — i.e., models shouldn't just learn to game the evaluation patterns they saw in training.
Related event: Rachel Thomas highlights research on LLM instruction-following failures(3 posts)→
More from Research
- Embodied First Exam benchmarks coding agents on 1,090 tasks across 44 robots — ManlingLi_ · 2026-10-23
- Anima Anandkumar on neural operators: AI weather models beat supercomputers — AnimaAnandkumar · 2026-10-23
- Scaling GLM-5.3-Flash training to 1M token context, 64x over Megatron baseline — AndrewLBeam · 2026-10-23
- Experimental boolean-logic LM: microsecond forward pass on one CPU thread, but painfully slow training — cephaloform · 2026-10-23
- AdaDex turns seconds of human hand motion into generalizable dexterous robot skills — mangahomanga · 2026-10-23
- ReAPO paper: conditioning retries on failures beats GRPO in RLVR reasoning training — jasondeanlee · 2026-10-23