Paper claims 0.77% training, but released checkpoint actually uses 6.31% of parameters
yoavartzi · x · 2026-07-23
Peer review and reproducibility problems are not new
The poster argues that these trust and reproducibility issues have existed for a long time and likely could have been observed 15 years ago as well. They say peer review is broken, but also note that these problems did not stop the scientific system from being a huge net positive overall.
In a reply, they add a concrete example from their analysis: one paper claimed its method trained only 0.77% of the base model’s parameters, presenting that as a headline advantage. But the checkpoint actually released trained 6.31% of parameters — about 8× more. The 0.77% figure only held under a narrower condition, which they say is just one example of the broader issue.
Related event: AI Paper Reproduction Failures Often Due to Missing Code(3 posts)→
More from Research
- Autoencoders compress data into latent spaces and flag anomalies by reconstruction error — burny_tech · 2026-07-23
- OpenAI incident and new paper show AI monitors still miss hidden sabotage — TheTuringPost · 2026-07-23
- 15 classic AI papers every engineer should read, from Transformers to RLHF — kalyan_kpl · 2026-07-23
- Tracking source IDs across edits could turn provenance into a falsifiable claim — tallmetommy · 2026-07-23
- 30 PyTorch problems cover MLSys inference, quantization, KV cache, and batching — kalyan_kpl · 2026-07-23
- NeurIPS workshop will focus on child safety, privacy, and synthetic-content risks in AI — chhaviyadav_ · 2026-07-23