Stop Reading the Hessian as a Matrix: Eigenvalues Are Local Curvature of the Loss Surface
techNmak · x · 2026-10-05
The author shares a geometric way to understand the Hessian: rather than a table of second derivatives, think of it as describing the local second-order geometry of the loss surface.
- Moving along any unit direction u at a point in parameter space, the second directional derivative of the loss is u^T H u
- For eigenvector vᵢ, vᵢ^T H vᵢ = λᵢ, so each eigenvalue gives the local curvature along that direction
- A near-zero positive eigenvalue means a flat direction; a large positive one means sharp bending
- This explains optimization difficulty: when curvature varies greatly across directions, a learning rate fine for flat directions can be too aggressive along sharply curved ones
More from Research
- Shanghai AI Lab Proposes SCALE: Entropy-Gated Control to Reverse SFT Features — Shanghai-AI-Laboratory · 2026-10-05
- UCVG.cpp: Generate Control Vectors for Any LLM From a Single Prompt Pair — Egor4more · 2026-10-05
- How many digital minds on one GPU cluster? Synthese paper probes interwoven AI consciousness — burny_tech · 2026-10-05
- New preprint argues 'adaptive reframing'—revising problem representations—is a key unstudied dimension of intelligence — ValerioCapraro · 2026-10-05
- A Beautiful 2D Embedding Is Not Quantitative Evidence, Warns AI-for-Science Researcher — bravo_abad · 2026-10-05
- ML Conference AC warns: papers that are 'incomprehensible' will be desk rejected — tyrell_turing · 2026-10-05