Surge AI's ComplexConstraints benchmark: frontier models score just 0–40%
StewartalsopIII · x · 2026-11-06
The screenshot features Edwin Chen (Founder, Surge AI) introducing ComplexConstraints, a new instruction-following benchmark testing entangled real-work constraints (scheduling with 18 interlocking conditions), covering conditional, planning, multistep and implicit constraints. Current models score only 0% to 40%.
Training a 4B model on 1,000 companion examples reached parity with a model 60x its size, with gains transferring to MultiChallenge and AdvancedIF (e.g. Meta AdvancedIF overall pass rate 28.15%→36.60%). The poster adds a war story of Opus triaging an upstream schema change correctly by the letter while ignoring compute efficiency — echoing the point that real instructions aren't lists of independent rules.
More from Models
- User screenshot suggests Anthropic's weekly Claude limits feel effectively unlimited — StewartalsopIII · 2026-11-06
- Leonie Shares Full Slide Deck on Building Decision Models, Still Early Days — helloiamleonie · 2026-11-06
- Liquid AI open-sources d1 decision models: a 2B model purpose-built for classification calls — helloiamleonie · 2026-11-06
- No RL needed: temperature scaling is the simple trick teams use to calibrate model confidence — helloiamleonie · 2026-11-06
- Freshly trained decision models are overconfident: claims 90% sure but only 68% correct — helloiamleonie · 2026-11-06
- Liquid AI lifts the decoder's causal mask: bidirectional attention beats causal for decision models — helloiamleonie · 2026-11-06