Perplexity details decider data hygiene: official splits only, deduped, 16.6% of one stage
antoine_chaffin · x · 2026-10-17
antoinechaffin details how DI training splits were included for fair comparison.
- Only official training splits, never test data; careful deduplication against every test set for all train sets, even non-official ones
- The data appears in only one specific training stage at 16.6% of that stage, 0% elsewhere
- The final checkpoint is a model soup chosen as the best compromise across all evals including many held-out sets, with DI never coming at the cost of any of them
More from Models
- Claude cracks 30-year mystery: ECDSA curve seeds were just RNG output — scaling01 · 2026-10-17
- KOL teases mystery model argon, says people are 'severely underestimating' it — iruletheworldmo · 2026-10-17
- Reddit post argues Anthropic has led benchmarks for 6+ months and is 2-3 iterations ahead internally — Concern-Excellent · 2026-10-17
- Perplexity traces decider v1.2 regressions to longer training, mitigates via model souping — antoine_chaffin · 2026-10-17
- Dual-mask fine-tuning recovers cacheability with only -0.4 DI vs full bidirectional — antoine_chaffin · 2026-10-17
- Perplexity open-sources next-gen decider models in 27B and 4B sizes with up to 5x faster inference — antoine_chaffin · 2026-10-17