Independent test: Liquid d1-ultra scores 77.1%, dethroning Perplexity's Decider v1.2 by 10 points
JosephJacks_ · x · 2026-10-21
Zeki Works published an independent benchmark of 25 decision models over 2,198 native-Turkish reading comprehension and grammar questions, challenging Perplexity's ClaimBench leaderboard claim.
Key findings
- Perplexity says pplx-decider-v1.2 tops ClaimBench at 67.1%; but on Zeki's same questions, Liquid d1-ultra — not listed on ClaimBench — scored 77.1%, a full 10 points ahead of Decider v1.2.
- No model led on every metric: d1-ultra won accuracy and estimated cost savings across all 10 LLM setups; the smallest model, Luna (Decisions API), had the best calibration gap.
- Decision models answered up to 92.1% of reading questions correctly but at most 70.7% of grammar ones — a clear gap between knowing a rule and applying it.
All questions and answers are public for verification.
More from Models
- Users hit a new Claude friction: paternalistic redaction of 'sensitive' data, no explanation — mattbeane · 2026-10-21
- Beam 1.0 tops Vals AI private eval as #1 US open-weight model, ranks #14 globally — bigblueboo · 2026-10-21
- Shanghai AI Lab releases Sage-A1.5, a 35B MoE agentic model for long-horizon tasks and scientific discovery — InternScience · 2026-10-21
- Theo Calls OpenAI Astra the Most Disappointing Release of the Year — bookwormengr · 2026-10-21
- Ex-Microsoft exec impressed by mysterious Argon model: strong math, tiny context — ivan_bezdomny · 2026-10-21
- Pangram detection clarified: AI-fixing your typos won't get flagged — soumitrashukla9 · 2026-10-21