Pleias team on leaderboards: build internal evals for real use cases, but can't ignore them
antoine_chaffin · x · 2026-10-13
Responding to a point that knowing use cases is 90% of the work for encoders, antoinechaffin agrees: Pleias builds internal benchmarks for tasks users actually care about and pays close attention to user needs—outsourcing that to benchmark makers makes no sense.
But he notes the reality that newcomers pick models off popular leaderboards, so ignoring them entirely isn't an option. His stance: never fit leaderboards if it degrades what users need, while actively giving benchmark makers feedback and eval data—not to bias them, but to contribute useful evaluation information.
More from Models
- Gemini 4 Argon pricing ($2/$10) spotted on Google's SKU page, global release imminent — gaganghotra_ · 2026-10-13
- A style guide for making Claude-like models write normally, and its linguistic traps — gleech · 2026-10-13
- OpenAI denies Pro 200 grandfathering to a subscriber billed through the eligibility window — IrrelevantNecessity · 2026-10-13
- autotrust's JEV-35B-NVFP4, a Qwen3.5 MoE-based decision model, hits HF trending — autotrust · 2026-10-13
- Theo benchmarks Opus 5.5, GPT-6.1 Sol and Muse Wonder Full on a bug fix — shadcn · 2026-10-13
- Claude Code at $100/month plus $20 ChatGPT: AI subscription costs stack up — dumpshoot · 2026-10-13