Depth scaling experiment: 4→64 ReLU layers yields only ~6pt accuracy gain
halvarflake · x · 2026-10-21
An experiment visualizes image reconstruction quality as a small ReLU network (width 16) scales from 4 to 64 layers on the famous Zidane volley photo, holding training pixels (25,000), minibatches, and seed fixed.
Key observations:
- Parameters grow from 1,136 (4 layers) to 17,456 (64 layers), but validation accuracy only improves from 66.4% to 72%, saturating after 24 layers (oscillating in the 71-72.5% band).
- Deeper networks mainly refine textures and edge details; the coarse structure is already captured by shallow models.
- Takeaway: for this class of small fitting tasks, depth shows sharply diminishing returns.
More from Research
- RFD4-Proteina's 'underrated' docs detail partial diffusion for protein redesign — AllThingsApx · 2026-10-23
- MPI-Sec PhD Thesis on Vulnerability Detection Benchmarks Includes Pre-LLM Classic "Top Score on the Wrong Exam" — mboehme_ · 2026-10-23
- LiveNerf's 29-day data finds no nerfing of Claude Opus 5.5 — TheOnlyVibemaster · 2026-10-23
- New paper makes 9 falsifiable predictions toward a major AI-enabled terror attack by 2027-2028 — StephenLCasper · 2026-10-23
- HumanoidPose recovers humanoid robots' full-body joint states from RGB video alone — cholz · 2026-10-23
- Sophont's Holon-S1 predicts genes and biomarkers from routine pathology slides — iScienceLuvr · 2026-10-23