Paper claims 0.77% training, but released checkpoint actually uses 6.31% of parameters
yoavartzi · x · 2026-07-23
Peer review and reproducibility problems are not new
The poster argues that these trust and reproducibility issues have existed for a long time and likely could have been observed 15 years ago as well. They say peer review is broken, but also note that these problems did not stop the scientific system from being a huge net positive overall.
In a reply, they add a concrete example from their analysis: one paper claimed its method trained only 0.77% of the base model’s parameters, presenting that as a headline advantage. But the checkpoint actually released trained 6.31% of parameters — about 8× more. The 0.77% figure only held under a narrower condition, which they say is just one example of the broader issue.
Related event: AI Advances Academic Peer Review and Reproducibility Checks(4 posts)→
More from Research
- Nature paper images cellular activity across all organs, revealing body-wide circuits — arjunrajlab · 2026-09-11
- Skyfall GS Uses Flux to Refine Gaussian Splatting, Accepted at ECCV 2026 — ducha_aiki · 2026-09-11
- Could 10k agents discover learning methods beyond backprop, or just tweak existing ones? — SeunghyunSEO7 · 2026-09-11
- Apodex Launches TRACES, First Benchmark for Evaluating 'Discoverative AI' on Real-World Problems — Faheem_uh · 2026-09-11
- TRACES grades the process, not the answer: six-dimension eval for open-ended AI science — Faheem_uh · 2026-09-11
- Apodex launches TRACES, a benchmark grading AI on open-ended discovery instead of known answers — Faheem_uh · 2026-09-11