Paper claims 0.77% training, but released checkpoint actually uses 6.31% of parameters
yoavartzi · x · 2026-07-23
Peer review and reproducibility problems are not new
The poster argues that these trust and reproducibility issues have existed for a long time and likely could have been observed 15 years ago as well. They say peer review is broken, but also note that these problems did not stop the scientific system from being a huge net positive overall.
In a reply, they add a concrete example from their analysis: one paper claimed its method trained only 0.77% of the base model’s parameters, presenting that as a headline advantage. But the checkpoint actually released trained 6.31% of parameters — about 8× more. The 0.77% figure only held under a narrower condition, which they say is just one example of the broader issue.
Related event: AI Advances Academic Peer Review and Reproducibility Checks(4 posts)→
More from Research
- NLI alone rejects 5/6 new agent memories; a 0.45 similarity gate fixes it all — Alternative_Pin9598 · 2026-10-26
- AI folks, stop saying "biology" when you mean drug development — KevinKaichuang · 2026-10-26
- ostris pre-training "Primal VAE": 1-channel latents match modern 8x VAEs via patching — ostrisai · 2026-10-26
- Shanghai AI Lab-led team releases 100-hour egocentric wearable dataset for humanoid training — ChuckDBrooks · 2026-10-26
- ConvFill: A 0.6B Talker Buys Time While a Large Reasoner Whispers the Real Answer — andimarafioti · 2026-10-26
- Flow Reasoning Models turn flow matching into iterative reasoners, hitting 99.5% on Sudoku-Extreme — alec_helbling · 2026-10-26