Perplexity traces decider v1.2 regressions to longer training, mitigates via model souping
antoine_chaffin · x · 2026-10-17
antoinechaffin on decider v1.2's known regressions and data handling.
- Hundreds of evals surfaced a few non-dramatic regressions; users are invited to report bad results
- Root causes traced not to DI data but to: (1) longer, broader training drifting the model from backbone capabilities; (2) side effects of big vision gains
- Both mitigated via model souping
- For fair comparison, official training splits of DI benchmarks were carefully included: never test data, deduped against every test set, only 16.6% of one training stage and 0% elsewhere; the final checkpoint is a model soup balanced across all evals including held-out sets
More from Models
- Claude cracks 30-year mystery: ECDSA curve seeds were just RNG output — scaling01 · 2026-10-17
- KOL teases mystery model argon, says people are 'severely underestimating' it — iruletheworldmo · 2026-10-17
- Reddit post argues Anthropic has led benchmarks for 6+ months and is 2-3 iterations ahead internally — Concern-Excellent · 2026-10-17
- Dual-mask fine-tuning recovers cacheability with only -0.4 DI vs full bidirectional — antoine_chaffin · 2026-10-17
- Perplexity open-sources next-gen decider models in 27B and 4B sizes with up to 5x faster inference — antoine_chaffin · 2026-10-17
- Perplexity ships pplx-decider v1.1: bidirectional attention adds +2.8 DI, new sizes 27B/4B — antoine_chaffin · 2026-10-17