Inverse IFEval tests flawed questions, code without comments, mid-chat rule changes
math_rachel · x · 2026-10-19
Entry 2 of Rachel Thomas's thread elaborates on Inverse IFEval's dimensions: it benchmarks the ability to follow out-of-distribution instructions, including inherently flawed questions, code without comments, not using bulleted lists, and instructions that change within an interaction — diagnosing biases baked in by SFT.
Related event: Rachel Thomas highlights research on LLM instruction-following failures(3 posts)→
More from Research
- Embodied First Exam benchmarks coding agents on 1,090 tasks across 44 robots — ManlingLi_ · 2026-10-23
- Anima Anandkumar on neural operators: AI weather models beat supercomputers — AnimaAnandkumar · 2026-10-23
- Scaling GLM-5.3-Flash training to 1M token context, 64x over Megatron baseline — AndrewLBeam · 2026-10-23
- Experimental boolean-logic LM: microsecond forward pass on one CPU thread, but painfully slow training — cephaloform · 2026-10-23
- AdaDex turns seconds of human hand motion into generalizable dexterous robot skills — mangahomanga · 2026-10-23
- ReAPO paper: conditioning retries on failures beats GRPO in RLVR reasoning training — jasondeanlee · 2026-10-23