Neural net series: multi-layer nets and why cross-entropy and squared error share the same gradient
alfcnz · x · 2026-09-24
alfcnz's from-scratch neural network series continues with multi-layer nets and the learning signal.
- C. Multi-layer net: one layer learns one fixed template per direction, but real roads vary; the hidden layer maps pixels to road features to steering, with nobody telling hidden units the right answer
- D. Learning signal: the loss L(w, x, y) = C(y, ỹ) is a single number for how wrong the net is (squared error on the board); cross-entropy is left out since with each output paired to its loss, the gradient with respect to s is ỹ − y either way
- Softening: σ(s) for hidden units, softargmax(s) at the output
More from Research
- Anthropic: Claude discovered a previously unknown enzyme system in phage DNA — KevinKaichuang · 2026-09-24
- TRACES: A New Benchmark That Grades AI Problem-Solving Process, Not Just Correct Answers — dr_cintas · 2026-09-24
- ECCV MMBU Benchmark Shows VLMs Answer Biomedical Questions Without Knowing What They See — davidjhwu · 2026-09-24
- Study: authors can reliably predict which of their papers will be highly cited — RexDouglass · 2026-09-24
- Applied Compute uses Jev to auto-cluster failure modes in RL training traces — rhythmrg · 2026-09-24
- New note extends 'The Economics of Recursive Self-Improvement' paper — CFGeek · 2026-09-24