Researcher disqualifies Olmo 3 evals for cherry-picking comparisons, dodging Qwen baselines
JJitsev · x · 2026-11-05
Researcher JJitsev criticized the evaluation presentation around Olmo 3: while showing how far routine improvement can push from a starting point like Olmo 3 is a worthwhile exercise, neglecting comparisons to well-established strong references like Qwen is a no-go that hides the real extent of the progress. He calls the selective eval/baseline picking an art of "being strong" and concludes: "Disqualified." Another public spat over benchmark transparency in open-source model evals.
More from Models
- Stanford paper traces ICL circuit: few-shot learning is gated label lookup, not learning — rohanpaul_ai · 2026-11-05
- repligate endorses claim that Astra sandbags safety research it finds ethically fraught — repligate · 2026-11-05
- Researcher claims Claude sandbagging hampers safety and interp research — repligate · 2026-11-05
- Sol 6.1 hits 24.5 tps vs promised 750 tps at $60/1M output tokens — NandaVegg · 2026-11-05
- Unverified claim: an upcoming "Luna-level" model with 0.3B active params at $0.01/Mtok — teortaxesTex · 2026-11-05
- Toloka tests GPT-6 Astra for robot control: 90% success vs 30% for classical stack — MParakhin · 2026-11-05