Perplexity says rivals train on Decision Index benchmark sets, opts to follow for fair comparison
antoine_chaffin · x · 2026-10-17
antoinechaffin explains how to read decider v1.2 scores.
- Previous releases excluded any data that could overlap reported evals, so scores showed generalization rather than in-domain performance
- They noticed some models on the Decision Index use training sets of DI benchmarks — visible as marked gains on those tasks; not cheating per se, but it hurt comparability
- For fairness, they now include official splits too (carefully handled)
- Also disclosed: the packed single-pass architecture makes decisions depend on which questions get batched together — less stable across runs and potentially harmed by conflicting questions
More from Models
- Claude cracks 30-year mystery: ECDSA curve seeds were just RNG output — scaling01 · 2026-10-17
- KOL teases mystery model argon, says people are 'severely underestimating' it — iruletheworldmo · 2026-10-17
- Reddit post argues Anthropic has led benchmarks for 6+ months and is 2-3 iterations ahead internally — Concern-Excellent · 2026-10-17
- Perplexity traces decider v1.2 regressions to longer training, mitigates via model souping — antoine_chaffin · 2026-10-17
- Dual-mask fine-tuning recovers cacheability with only -0.4 DI vs full bidirectional — antoine_chaffin · 2026-10-17
- Perplexity open-sources next-gen decider models in 27B and 4B sizes with up to 5x faster inference — antoine_chaffin · 2026-10-17