Optimization for Data Science · Lecture 9
Gradient descent avoids strict saddles
A strict saddle repels along at least one direction; it may attract along others. Explore the exceptional starting points, the separation that drives the proof, and what can fail when the step is too large.
1 The quadratic saddle from the lecture
Consider the two-dimensional function from the notes. The negative-curvature direction pushes away from the origin, while the positive-curvature direction pulls toward it. Only a “small” set gets trapped at zero.
The phase portrait from the notes
Blue curves and arrows show continuous gradient flow. Colored dots show discrete GD. Click to choose an initial point.
The quadratic in 3D
The z direction curves upward; the w direction curves downward. Drag to rotate, or use the view control below.
Height is rescaled to fit the current curvatures. Paths are overlaid for visibility.
The calculation and the “small” exceptional set
The gradient vanishes at (0, 0), and the Hessian is diag(−α, β). Thus the origin is a strict saddle, with an unstable horizontal direction and a stable vertical direction. Here L = max{α, β}; the controls keep 0 < η < 1/L.
wk = (1 + ηα)kw₀, zk = (1 − ηβ)kz₀.
Since 1 + ηα > 1 and 0 < 1 − ηβ < 1, GD converges to (0, 0) exactly when w₀ = 0. That stable line has area zero, so a random initialization with a density lands on it with probability zero.
The continuous-time picture in the notes follows the gradient flow:
ẇ = αw, ż = −βz, w(t) = eαtw₀, z(t) = e−βtz₀.
The blue trajectories use these exact flow solutions; their arrows indicate direction. The colored dots use the GD recursions above. Both dynamics have the same stable line, although their trajectories generally differ.
This quadratic is unbounded below. Off the stable line, escape means |wk| → ∞ and f(wk, zk) → −∞; there is no minimum to reach. The next experiment shows a landscape with minima as well as saddles.
2 Many saddles, exceptional starts
Run a cloud of random initializations on the same surface. Then put the highlighted point exactly on a stable line: this deliberately chosen start can converge to a saddle.
Trajectories on the contours
Click to choose the highlighted start. Coordinate fields provide the same control with a keyboard.
The same trajectories in 3D
Height is f(x, y). Drag to rotate the surface, or use the view control below.
Trajectories are overlaid for visibility.
Why this example fits the theorem
The Hessian is diag(−cos x, −cos y), so its operator norm is at most 1 everywhere. The four visible saddles are (0, ±π) and (±π, 0), with eigenvalues −1 and 1. The minima are (±π, ±π). The origin is a local maximum.
Within this window, the saddle basins are the two coordinate axes with the origin removed. Each is one-dimensional and has area zero. On x = 0, the x update stays exactly zero; for a nonzero x near zero, the update pushes it away. The maximum is also an unstable stationary point.
The random cloud is a finite simulation, not a proof of probability zero. The claim follows here from the explicit stable sets. The four saddle events form a finite union of null events.
3 The mechanism behind the proof
Zoom in on the saddle (0, π). Start two trajectories with the same stable coordinate and a tiny difference in the unstable coordinate. They cannot both remain in a small neighborhood forever.
Stable and unstable directions
Coordinates u = x and v = y − π. The circle has radius 0.6. The two trajectories have identical v coordinates.
Separation grows before escape
Linear iteration axis · logarithmic separation axis. The dotted curve is the local linearization, shown until it reaches 0.6.
Connect the picture to the cone argument and Fubini
At the saddle, the derivative of the GD map is diag(1 + η, 1 − η). Within the radius-0.6 ball, the Hessian differs from its value at the saddle by at most 1 − cos(0.6) < 1/4. These are the constants used in the lecture proof.
For two points, write U = |Δu| and V = |Δv|. If U ≥ V, the cone argument gives U⁺ ≥ (1 + η/2)U and U⁺ ≥ V⁺, as long as both points stay in the ball. Here V is exactly zero at every step, so the separation is particularly easy to see.
Two distinct trapped points therefore cannot have the same stable coordinate v. In the general proof, this makes the trapped set a graph over a subset of the stable/center coordinates; its slices in the unstable directions contain at most one point. Fubini gives measure zero. The proof needs this geometry, not a full smooth manifold theorem.
For ηL < 1, ‖T(a) − T(b)‖ ≥ (1 − ηL)‖a − b‖. Pulling the local null set back through any number of GD steps preserves measure zero. A convergent orbit eventually stays in the ball, so a countable union finishes the proof.
A smaller δ delays escape. The local approximation predicts about log(0.6/δ) / log(1 + η) iterations; this illustration supplies no uniform escape-time guarantee for the general theorem.
4 A larger step can collapse an entire region
The step-size restriction protects the global part of the proof. Here the origin still repels locally, but at η = 1.5 an open set maps exactly onto its stable line in one step.
Positive area becomes a line
Starts are uniform on 1.05 < |x₀| < 1.7, |y₀| < 1.1. Shading marks |x| ≥ 1. Try one Step before Run.
The one-dimensional GD map
Outside [−1, 1], the map is flat at zero when η = 1.5. Close to zero its slope is 1 + η/3 > 1.
The function and the exact counterexample
φ(x) = −x²/6 + 2|x|³/3 − x⁴/4 if |x| ≤ 1;
φ(x) = x²/3 − 1/12 if |x| ≥ 1.
The two pieces match through their second derivatives. Inside [−1, 1], φ″(x) = −1/3 + 4|x| − 3x²; outside, φ″ = 2/3. Thus |φ″| ≤ 1 and F has a globally 1-Lipschitz gradient. At the origin the Hessian is diag(−1/3, 1/3).
η = 3/2 ⇒ T(x, y) = (3x(1 − |x|)+²/2, y/2).
For every |x₀| ≥ 1, x₁ = 0 exactly, and subsequent iterates tend to (0, 0). The minima are (±(1 − √(2/3)), 0). Even though η = 1.5 < 2/L gives descent, it does not give saddle avoidance.
On these outer strips, det DT = 0: the map loses a dimension. This is precisely where the null-preimage argument fails. At η = 1.49 the outer strips do not collapse, and the sampled trajectories eventually reach the minima.
The code evaluates the algebraically equivalent update x⁺ = (1 − 2η/3)x + ηx(1 − |x|)+². This preserves the exact zero at η = 1.5 without adding any snapping rule.