Manual checks of 105 papers find most reproducibility failures come from missing code and mismatched results
ChenhaoTan · x · 2026-07-23
Manual checks across 105 papers find reproducibility failures are usually mundane, not mysterious
The author says their analysis suggests that the main reasons papers fail to reproduce are often not model capability issues. Instead, the problems are usually practical and procedural.
They give one concrete example: a paper claims its method trains only 0.77% of the base model’s parameters, but the checkpoint actually released trains 6.31%—about 8× more. The smaller figure only works if you assume an adapter one-eighth the size of the one that was shipped.
The attached chart summarizes the most common causes of non-reproduction in their sample of 105 papers:
- Code doesn’t run as shipped: 58
- Numbers don’t match the paper: 48
- Data not available: 42
- No runnable code released: 38
- Trained models not released: 18
- Code doesn’t match the paper: 15
- Relies on legacy or unavailable models: 4
The takeaway: 101 of 105 papers had at least one of these issues, so reproducibility is often blocked by missing artifacts, mismatched code, and inconsistent reporting rather than by raw model capability.
Related event: AI Advances Academic Peer Review and Reproducibility Checks(4 posts)→
More from Research
- New review: how prefrontal cortex dynamics turn world models into flexible behavior — AnnaCiaunica · 2026-10-27
- LIBERO-MAX: Robot Policies Drop 11–26 Points When the World Changes Mid-Task — drmapavone · 2026-10-27
- Kolab Benchmark Uses Pro Screen Recordings to Show AI Agents Overestimate Collaboration Quality — minnesotanlp · 2026-10-27
- One BBQ Example Pushes LLM Fairness Scores Past 90%, EMNLP Paper Questions Benchmarks — radamihalcea · 2026-10-27
- Real2Sem2Real turns one phone video into 2,000+ robot demos, matching 150 teleop demos — Scobleizer · 2026-10-27
- Understanding MuonH (Hyperball) through weight norm geometry of scale-invariant networks — jasondeanlee · 2026-10-27