Optimization for Data Science · Lecture 6 demo

Preconditioning and Newton's method

Four short experiments that follow the board notes, meant to be opened one at a time. As in class, preconditioned gradient descent is wk+1 = wk − η P−1∇f(wk), which minimizes the linear model plus the anchor ‖z − wk‖P2/(2η). Nothing runs until you press Run; click a plot to move the starting point.

1Preconditioning rescales directions

On the quadratic f(w) = ½ wᵀAw with A = UDUᵀ, gradient descent crawls along the flat direction of the valley. A preconditioner P changes the norm in which steps are measured. In the coordinates z = P1/2w the same run is plain gradient descent on a function whose condition number is κ(P−1/2AP−1/2).

Condition number κ(A)
1101001000
Valley angle θ
Preconditioner
Power α in P = Aα
Iterationk = 0

Original coordinates w

Contours of f. Click to choose the starting point.

GD, η = 1/LPGD, η = 1/LPanchor ball ‖z − wk‖P = r

New coordinates z = P1/2w

Contours of φ(z) = f(P−1/2z). Click to choose the starting point.

GD on φP1/2wk for the PGD iterates

Objective

f(wk)/f(w0), logarithmic scale. Dashed: the theorem's bounds (1 − μ/L)k and (1 − μP/LP)k.

GDPGD
κ(A)
κP = κ(P−1/2AP−1/2)
GD iterations to f/f0 ≤ 10⁻⁶
PGD iterations to f/f0 ≤ 10⁻⁶
Diagram check: maxk ‖P1/2wk − zk‖
Try this
  1. Press P = I and run. Gradient descent kills the steep direction at once and then crawls along the valley.
  2. Move α from 0 to 1. With P = Aα the condition number becomes κP = κ1−α. At P = A, PGD reaches the minimizer in one step, but that step requires solving a linear system with A.
  3. Press Jacobi and change the valley angle. A diagonal preconditioner is perfect when the valley is aligned with an axis (θ = 0° or 90°) and does nothing at θ = 45°: it fixes scaling, not rotation.
  4. Compare the two panels. The hollow circles on the right are the PGD iterates mapped by P1/2; they sit on the gradient descent path for φ. This is the commutative diagram from the board.
  5. With Show anchor balls, the dashed ellipse is {z : ‖z − wk‖P = r} through the next iterate. The next iterate is the point of that ellipse with the smallest value of the linear model. For GD the ball is a circle.
Mathematical setup

f(w) = ½ wᵀAw,  A = U diag(1, 1/κ) Uᵀ,  wk+1 = wk − η P−1Awk.

The valley floor, the eigenvector with eigenvalue 1/κ, makes the angle θ with the horizontal axis. With M = P−1/2AP−1/2, the constants are LP = λmax(M) and μP = λmin(M), and PGD uses η = 1/LP. GD is the case P = I, with η = 1/L = 1. In z = P1/2w, the function φ(z) = f(P−1/2z) = ½ zᵀMz has gradient P−1/2∇f(w), so GD on φ from z0 = P1/2w0 produces zk = P1/2wk. The dashed lines in the chart are the bounds of the convergence theorem; on a quadratic the actual decrease is faster.

2Newton's method does not care about coordinates

Run Newton's method and gradient descent on a smooth convex function f and on ψ(z) = f(Az), a stretched and rotated copy. Newton's iterates on ψ are exactly A−1 times its iterates on f. Gradient descent's are not, and it slows down when the stretch makes ψ badly conditioned.

Stretch s in A = R(θ) diag(s, 1/s)
0.215
Rotation θ

The line search starts with t = 1 and halves t until f decreases enough. Gradient descent always uses it. Turning it off makes Newton take full steps, t = 1.

Iterationk = 0

Coordinates w: minimize f

Click to choose w0.

Newton on fGD on f

Coordinates z: minimize ψ(z) = f(Az)

Click to choose z0 = A−1w0.

Newton on ψA−1(Newton on f)GD on ψA−1(GD on f)

Objective gap

f(wk) − f⋆ and ψ(zk) − ψ⋆, logarithmic scale. The two Newton curves coincide.

Newton on fNewton on ψGD on fGD on ψ
Newton: maxk ‖Azk − wk‖
GD: maxk ‖Azk − wk‖
Newton iterations to gap ≤ 10⁻¹⁰ (f / ψ)
GD iterations to gap ≤ 10⁻¹⁰ (f / ψ)
κ of the Hessian at the minimizer (f / ψ)
Try this
  1. Run. After a few damped steps, Newton's gap drops like 10⁻², 10⁻⁶, 10⁻¹²: the number of correct digits doubles at every step.
  2. Increase the stretch or change the rotation. The condition number of ψ grows and GD on ψ (dashed teal) needs many more iterations, while the hollow teal circles, A−1 times GD on f, drift away from its path. Newton's two curves stay on top of each other.
  3. Set the stretch to 1 and the rotation to 0: then A = I and everything coincides.
  4. Turn off the line search and click far from the minimizer. Pure Newton can take wild steps, yet its iterates on f and on ψ still correspond exactly.
Mathematical setup

f(w) = ew₁+3w₂−0.1 + ew₁−3w₂−0.1 + e−w₁−0.1,  w⋆ = (−½ ln 2, 0),  f⋆ = 2√2 e−0.1.

This is the two-dimensional example of Boyd and Vandenberghe, Convex Optimization, Chapter 9. Newton's direction is d = −∇²f(w)−1∇f(w) and GD's is d = −∇f(w). Both use the step w + td, where t starts at 1 and is halved until f(w + td) ≤ f(w) + ¼ t⟨∇f(w), d⟩. Since ∇ψ(z) = Aᵀ∇f(Az) and ∇²ψ(z) = Aᵀ∇²f(Az)A, Newton's direction for ψ is A−1 times its direction for f. The line search compares only function values and ⟨∇f, d⟩, which do not change, so the damped method is affine invariant too. GD's direction for ψ is −Aᵀ∇f(Az), which corresponds to the step −AAᵀ∇f(w) in the original coordinates: a different method.

3Newton looks for critical points, not minimizers

Each Newton step jumps to the critical point of the second-order Taylor model. When f″(xk) > 0 the model is convex and the step goes to its minimizer. When f″(xk) < 0 it goes to its maximizer. The method sees only f′/f″, so on f and on −f it produces the same iterates.

Starting point x0
Function
Iterationk = 0

The function and its quadratic model at xk

Click to choose x0. The strip below colors every starting point in [−2.2, 2.2] by the critical point that Newton reaches.

current functionquadratic model at xklimit −1limit 0limit +1no limit in 200 steps
xk
f″(xk) of the current function
The step goes to the model's
Newton converges to
Try this
  1. Run from x₀ = 0.3. The model is concave, Newton jumps to its maximizer, and the iterates converge to the local maximum at 0.
  2. Press −f. The curve and the model flip upside down, but the iterates do not move: Newton cannot tell a minimizer from a maximizer.
  3. Move x₀ toward 1/√3 ≈ 0.577, where f″ = 0. The model's critical point runs off to infinity. In the strip, the colors interleave on finer and finer scales near ±1/√3.
  4. Start at x₀ = 1.8. Newton converges to the minimizer at 1, and near the end the number of correct digits doubles at each step.
Mathematical setup

f(x) = x⁴/4 − x²/2,  f′(x) = x³ − x,  f″(x) = 3x² − 1,  xk+1 = xk − f′(xk)/f″(xk) = 2xk³/(3xk² − 1).

The critical points are −1 and 1 (minimizers of f) and 0 (a local maximizer). Replacing f by −f changes the signs of f′ and f″ together, so the ratio and the iterates are unchanged. The strip runs up to 200 Newton steps from each starting point and colors it by the critical point it converges to.

4Newton fractals

Let f(x, y) = Re G(x + iy), where G is a polynomial with G′ = p. Newton's method for f is exactly Newton's root-finding method zk+1 = zk − p(zk)/p′(zk) in the complex plane. The critical points of f are the roots of p, and every one of them is a saddle point of f. Each starting point is colored by the critical point Newton reaches, and darker means more steps.

Polynomial p = G′

Iteration budget

Basins of attraction

Click a point to draw the Newton path from it; zooming centers on the last point clicked. White circles: the critical points of f, all saddles. Black: no convergence within the iteration budget.

Startclick the picture
Result—
View
Try this
  1. With z³ − 1, zoom in on a boundary between two colors. Every neighborhood of a boundary point contains starting points that go to each of the three saddles.
  2. Click points in different basins. Newton converges, often in a handful of steps, but always to a saddle point: f has no minimizers at all.
  3. Switch to z³ − 2z + 2 and click inside one of the black regions. What do the iterates do?
  4. Increase the iteration budget. Black pixels near basin boundaries become colored (they were just slow), but the large black regions do not change.
Mathematical setup

Write z = x + iy and p = G′. Since ∂f/∂x = Re p and ∂f/∂y = −Im p, the gradient of f, read as a complex number, is \overline{p(z)}. Similarly, the Hessian maps a direction v = v₁ + iv₂ to \overline{p′(z)v}. The Newton system ∇²f(w)d = −∇f(w) therefore reads p′(z)d = −p(z), which gives the complex Newton step. The Hessian is [[Re p′, −Im p′], [−Im p′, −Re p′]]. It has trace 0 and determinant −|p′|², so every critical point with p′ ≠ 0 is a saddle. A pixel is assigned to a root once an iterate comes within 10⁻⁶ of it. It is black if this does not happen within the budget, if p′ vanishes, or if the iterates leave every bounded region.