Mistral researcher: randomize everything when evaluating decision models
antoine_chaffin · x · 2026-10-31
Antoine Chaffin, working on decision models at Mistral, warns that evaluations contain many exploitable signals that don't hold on real traffic — so randomize everything when evaluating decider models. A commenter notes his meticulousness ensures knowledge trickles down to open source and spurious techniques don't mislead people.
More from Models
- Cohere x Palantir model benchmark mocked online: 670 tokens/s but questionable output — XiRw · 2026-10-31
- Inference optimization race heats up: practitioners now want fast, cheap 'good enough' models, not the absolute best — bendee983 · 2026-10-31
- OpenAI tightens Codex account limits: stacking paid accounts on one device can now get you locked out — TheZachMueller · 2026-10-31
- NVIDIA Releases Nemotron 3.5 Bolt E4B: 4GB On-Device Model with 80% Fewer Output Tokens — heatOverflower · 2026-10-31
- Fable 5.5 Max Output Stuns With Surprisingly Clean SVG Generation — Hesamation · 2026-10-31
- Pickover Asks Gemini to Draw Bathroom Plumbing Diagram, Invites You to Spot Errors — pickover · 2026-10-31