Can't reproduce a published baseline with its own code: how should a paper report it?
InsectActive95 · reddit · 2026-11-03
An ML researcher reports that rerunning a published baseline method with the authors' own released code and exact hyperparameters matches on most benchmarks but falls several points short on two, consistently across three seeds and four attempts months apart. They plan to report both published and reproduced numbers, but fear reviewers will trust the published figures and assume a flawed reproduction.
Key discussion points:
- What makes a reproduction convincing: unchanged author code, original hyperparameters, matching results elsewhere as sanity check, multiple seeds.
- Whether to contact the authors or file a GitHub issue first, and whether to mention that in the paper.
- How to word the results so they don't read as an attack while not looking weak.
- Do reviewers accept "published vs. reproduced" rows in main tables, or relegate them to the appendix?
More from Research
- Follow-up: Astra's weight edits only worked on Qwen's visible flaw, not on Gemma 270m — teortaxesTex · 2026-11-03
- EMBER: pretrained gene-centric foundation model predicts multigene perturbation effects — jmuiuc · 2026-11-03
- LLM edits another LLM: tweaking 0.016% of weights lifts Qwen3.5-0.8B math from 7.9% to 48.1% — teortaxesTex · 2026-11-03
- New preprint RLCE: reinforcement learning with comparative evidence for social intelligence — pliang279 · 2026-11-03
- Tau Ceti uses AI plus Lean kernel to formalize 'all of basic mathematics', open to contributors — AlexKontorovich · 2026-11-03
- New research links AI companions to less loneliness and higher self-esteem — serialchilla91 · 2026-11-03