Author confesses: 34-point AI hardware repair lead was an artifact, drops to zero after fair tuning
Connect_Agency_3975 · reddit · 2026-10-21
A preprint author openly dissects his own result: a memristor-crossbar repair controller seemed to beat a rival by 34.0 points, but after tuning the rival's learning rate and matching retraining steps the lead fell to 1.06, and an 8-seed preregistered replication gave -0.005 (95% CI -1.7 to +1.7) — a declared failure. At least 97% of the original lead came from an untuned baseline and unequal compute. Still holds: D beats retraining-only by 3.78 points at equal steps; forward-only BatchNorm refit stays near chance. Limits include simulation-only, test-set-picked learning rates, and sequential seed addition.
More from Research
- RFD4-Proteina's 'underrated' docs detail partial diffusion for protein redesign — AllThingsApx · 2026-10-23
- MPI-Sec PhD Thesis on Vulnerability Detection Benchmarks Includes Pre-LLM Classic "Top Score on the Wrong Exam" — mboehme_ · 2026-10-23
- LiveNerf's 29-day data finds no nerfing of Claude Opus 5.5 — TheOnlyVibemaster · 2026-10-23
- New paper makes 9 falsifiable predictions toward a major AI-enabled terror attack by 2027-2028 — StephenLCasper · 2026-10-23
- HumanoidPose recovers humanoid robots' full-body joint states from RGB video alone — cholz · 2026-10-23
- Sophont's Holon-S1 predicts genes and biomarkers from routine pathology slides — iScienceLuvr · 2026-10-23