Gradient Descent

Augustin-Louis Cauchy, 1847

Cauchy introduced gradient-based descent for systems of equations around 1847. This schematic shows constant learning-rate steps on an elliptical quadratic bowl—the contour lines are level sets of L(x,y), gold polyline traces iterates from a hot start toward the analytic minimum.