Optimization for Data Science · Lecture 3

Fast convergence. Which solution?

Two observations, three parameters, infinitely many exact fits. Compare gradient descent from zero with a different initialization to see what the algorithm learns—and what it preserves.

f(w) = ¼ ‖Xw − y‖²PL with μ = 1/κL = 1Hessian eigenvalues: 1, 1/κ, 0
1.00 / L
2.00
Change the row-space initialization

The teal run always starts at zero. The other starts at w₀ = a₀u₁ + b₀u₂ + c₀v. Change a₀, b₀ to change the transient behavior; only c₀ affects the limit.

Changing the model or initialization restarts both runs. κ measures curvature on the row space; the full Hessian is singular.

Parameter space

3D · drag to rotate

Orthonormal coordinates w = au₁ + bu₂ + cv. The gold line contains every exact fit; its intersection with c = 0 is the minimum-norm solution.

Zero startChosen startAll interpolators

Open circles: starts · filled dots: current iterates · diamonds: limits. The gray plane is the row space, c = 0. No loss height is shown.

Objective convergence

Logarithmic scale

PL gives geometric decay of the loss even though the objective is not strongly convex in all three parameters.

Zero startChosen startPL bound
Relative loss versus iterationThe relative objective is compared with the PL bound (1 minus eta divided by kappa) to the power t. Exact zeros appear on a separate row.

f(wt) / f(w₀) ≤ (1 − ημ)t = t

Zero initialization

w₀ = 0

Training loss
Parameter norm
Null coordinate ct
w∞ = w† · minimum-norm interpolator

Chosen initialization

Training loss
Parameter norm
Null coordinate ct
w∞ = w† + c₀v · closest interpolator to this start

The model and the two mathematical guarantees

A concrete least-squares problem. Use the orthonormal basis

u₁ = (1, −1, 0) / √2,
u₂ = (1, 1, −2) / √6,
v = (1, 1, 1) / √3.

With two observations and three parameters, set

X = √2 [u₁ᵀ; u₂ᵀ / √κ],   y = √2 (1, 1 / √κ).

Thus the plotted objective is exactly

f(w) = ¼ ‖Xw − y‖²
= ½ [(a − 1)² + (b − 1)² / κ].

The solution set is {u₁ + u₂ + cv : c ∈ ℝ}. Its minimum-norm point is w† = u₁ + u₂, with norm √2. Changing κ changes X and y together, preserving this solution set.

Fast convergence without strong convexity. The loss is flat along v, but

½ ‖∇f(w)‖² = ½ [(a − 1)² + (b − 1)² / κ²]
≥ (1 / κ) f(w).

Hence μ = 1/κ and L = 1. For 0 < η ≤ 1/L, the usual PL theorem gives f(wt) ≤ (1 − ημ)tf(w₀). “Linear convergence” means geometric decay per iteration.

Implicit bias. The GD recurrence separates into three coordinates:

at = 1 + (1 − η)t(a₀ − 1),
bt = 1 + (1 − η/κ)t(b₀ − 1),
ct = c₀.

The first two coordinates converge to 1; the null-space coordinate never changes. Therefore w∞ = X†y + Πker Xw₀. Zero initialization selects the minimum Euclidean norm. A nonzero null component selects a different interpolator with identical training predictions.

The curves use the exact GD recurrence above. Thin, faint paths show the upcoming trajectory; solid paths show the iterations already taken. The PL bound is shown over the full horizon. Points below 10⁻⁸ are clipped to the plot floor, while exact zero has a separate row. A run that starts with zero loss has no normalized curve. Minimum norm is a statement among interpolators; it is not a general guarantee of better test error.