German Evaluations Questioned for Skewing Results
JJitsev · x · 2026-07-16
The author points out that Soofi's evidence for being stronger in German also comes from the same batch of rewritten German evaluations. In this case, even if German is added to the training data, it might naturally hold an advantage, making the "stronger" claim unstable.
Overall conclusion: overhyped promotion should stop, otherwise it will damage trust in German/EU outputs and open-source releases.
Related event: SOOFI benchmark claims challenged over leakage and baseline reporting(13 posts)→
More from Models
- Kimi K3.1 Pro regresses on Debate Benchmark while DeepSeek 1017 gains at lower cost — teortaxesTex · 2026-10-27
- 4 Hours of LLM Startup Research Produced a Convincing but Nonsense Product Thesis — annetgriffin · 2026-10-27
- Dev argues Claude burns far more tokens than GPT models for the same coding work — haider1 · 2026-10-27
- MiniMax matches Claude Opus 4.5 in non-benchmaxxed evals, users report — teortaxesTex · 2026-10-27
- Real-world math: GPT-6.1 Terra's half price is offset by 2-3x token usage, costing more than Sol — lordpuddingcup · 2026-10-27
- Fable 5.5 rumored to ship this week as partners still test — kimmonismus · 2026-10-27