Paper claims 0.77% training, but released checkpoint actually uses 6.31% of parameters
yoavartzi · x · 2026-07-23
Peer review and reproducibility problems are not new
The poster argues that these trust and reproducibility issues have existed for a long time and likely could have been observed 15 years ago as well. They say peer review is broken, but also note that these problems did not stop the scientific system from being a huge net positive overall.
In a reply, they add a concrete example from their analysis: one paper claimed its method trained only 0.77% of the base model’s parameters, presenting that as a headline advantage. But the checkpoint actually released trained 6.31% of parameters — about 8× more. The 0.77% figure only held under a narrower condition, which they say is just one example of the broader issue.
Related event: AI Advances Academic Peer Review and Reproducibility Checks(4 posts)→
More from Research
- 'Vibe verification': ~60% of AI-written formal specs prove the wrong property — basedjensen · 2026-10-26
- Ito: a browser that detects browsing intent and auto-organizes tabs into Flows — andrewkuznet · 2026-10-26
- Cornell study says brains process language as linear sequences, challenging Chomsky's hierarchy — anirbanbandyo · 2026-10-26
- Hi3D V1.0 open-sourced with 2M training assets and 90% slicing success rate — kimmonismus · 2026-10-26
- Ortet evaluates 34 open-weight models on 50 health tasks, finds general leaderboards insufficient — elmanmansimov · 2026-10-26
- HackBench injects reward hacks into agent trajectories to stress-test monitors — DanielKhashabi · 2026-10-26