Gradient, Jacobian, and Hessian Explained in One Go
techNmak · x · 2026-07-16
This in-depth post uses a unified framework to explain the three most common differentiation objects when training AI models: - **Gradient**: The first derivative of a scalar function, answering "which direction changes the fastest." In optimization, gradient descent involves moving in the opposite direction of the gradient. - **Jacobian**: The first derivative matrix of a vector-valued function, describing how input changes affect multiple outputs. In automatic differentiation, it corresponds to the forward-mode Jacobian-vector product and the reverse-mode vector-Jacobian product. - **Hessian**: The second derivative matrix of a scalar function, describing curvature. At critical points, it can be used to determine whether a point is a local minimum, maximum, or a saddle point. The author concludes with an intuitive summary: the gradient shows direction, the Jacobian shows sensitivity, and the Hessian shows curvature; essentially, the Hessian is the Jacobian of the gradient.
More from Research
- Knowledgeless Language Models cut closed-book recall by anonymizing entities during pretraining — gdm3000 · 2026-07-21
- CPU-native LLM pilot passes 4 of 5 gates, but cross-tokenizer distillation still loses — WildPino25 · 2026-07-21
- A GPT 5.6 Sol workflow reportedly generates an infinite family of counterexamples — OwariDa · 2026-07-21
- A research guide v7 surfaces two contradictions instead of smoothing them over — Fantastic_Aside6599 · 2026-07-21
- Agents can remember facts, but still forget how to do the job — No_Advertising2536 · 2026-07-21
- AI-assisted search finds small counterexamples to the Gaussian Moments Conjecture — RichmanRonald · 2026-07-21