Optimization for Data Science · Lecture 8
Gauss–Newton and damping
Linearize the residual, then solve a least-squares problem. See why this can be extremely fast near a solution, why a full step can fail, and how Levenberg–Morrison–Marquardt (LMM) controls the step.
1When the linear model asks for too much
Use the example from the notes: r(w) = w² − 1. Near zero, its derivative is tiny. Canceling the linearized residual can require an enormous step, even though the two solutions are close by.
pGN = −r(w)/J(w), pLMM = −J(w)r(w)/(J(w)² + λ).
The default view stays near the roots. Arrows mark proposals outside the plot.
Linearize the residual
The tangent is r(w) + J(w)(z − w). GN finds its zero.
Minimize the local model
The local models match f at w. They need not bound f from above.
| Method | Proposed z = w + p | Step p | Actual f(z) | f(z) / f(w) |
|---|
The curvature we keep and the curvature we drop
The full Hessian is f″(w) = 6w² − 2. Gauss–Newton retains J(w)² = 4w² and drops r(w)r″(w) = 2(w² − 1). The dropped term vanishes at w = ±1. At w = 0, the true objective has negative curvature, but its GN model is flat. The inverse formula is undefined there; the minimum-norm convention would choose a zero step. Positive damping also gives a zero step, so damping alone does not escape this local maximum.
- At w = 0.1, compare the predicted steps with their actual objective values.
- Select Almost singular, then include both steps in the view.
- Select Near a solution and decrease λ. The damped step approaches the GN step.
2Fast local convergence, and the cost of damping
Start all three methods at the same point on f(w) = ½(w² − 1)². Fixed damping leaves a linear error term. Damping proportional to the residual vanishes near a root and recovers quadratic convergence.
Residual LMM uses λₖ = c |wₖ² − 1|.
Distance to the limiting root
Linear iteration axis · logarithmic error axis · no fitted convergence curves
| At the displayed iterate | wₖ | eₖ = |wₖ − w⋆| | Next λₖ | eₖ₊₁ / eₖ | eₖ₊₁ / eₖ² | Status |
|---|
For a positive start, w⋆ = 1; for a negative start, w⋆ = −1. At zero the distance to either root is 1. Ratios use the next computed iterate; they are hidden when eₖ < 10⁻⁷ or the next error < 10⁻¹³. Values below 10⁻¹⁴ lie on the chart floor.
Connect the curves to the theorem
For w > 0, set e = w − 1. The exact GN recursion is e₊ = e²/(2w). Thus e₊/e² tends to ½. For LMM, e₊ = [λe + 2we²]/(4w² + λ). A fixed λ > 0 gives an asymptotic error ratio λ/(4 + λ).
With λ = c |r(w)| and r(w) = (w − 1)(w + 1), damping is of the same order as the parameter error near a root. Both terms in the LMM recursion are then quadratic. The default c = 0.5 avoids an accidental higher-order cancellation that occurs for c = 1 when approaching 1 from below.
All methods take full steps with no line search. This is a local convergence illustration: a distant starting point can give a large increase in the objective. The computation stops at a parameter error of 10⁻¹⁴ or after 80 steps; numerical and singularity stops are reported explicitly.
3Fit a signal with many local minima
Fit the frequency and phase of a sinusoidal signal. The loss has many valleys: a curve can look plausible and still have the wrong frequency. Compare how the three methods behave from the same starting point.
n = 81 observations, d = 2 parameters. Residuals and Jacobians include the normalization 1/√n.
Presets reset noise to 0, λ to 0.1, and c to 0.5.
The fitted curves
Points are observations. Colored curves use the current parameters.
The loss landscape
Click the loss contours to choose a start; sliders are the keyboard alternative. The phase is periodic.
Optimization progress
| At the displayed iterate | ωₖ | φₖ | f(wₖ) | ‖rₖ‖ | ‖∇f(wₖ)‖ | Next λₖ | Status |
|---|
- Select Different basins and run: GN finds the wrong frequency; both damped methods recover the signal.
- Select Wrong frequency: all three approach a bad local minimum, even with noise turned off.
- Select Near the solution and switch the vertical scale to logarithmic to see fast local convergence.
Data, algorithms, and local minima
The 81 observations are generated from yᵢ = sin(2.4tᵢ + 0.4) + noiseᵢ at evenly spaced tᵢ in [0,10].
A fixed seed gives bounded, centered noise; the slider sets its maximum absolute value. These are synthetic observations. With noise, the generating parameters need not minimize the sample loss.
In the noiseless sinusoidal problem, the best loss is zero. The wrong-frequency preset approaches a positive-loss local minimum near (3.2591, −0.7311), with f ≈ 0.39856. Its full Hessian has positive eigenvalues, approximately 0.0644 and 7.1577. Thus the plateau is a bad local minimum, not just slow progress. The different-basins preset sends GN to another local minimum with f ≈ 0.37155. These outcomes use the preset damping values; changing them can change the destination.
All methods use the same residuals and starting point. GN solves (JᵀJ)p = −Jᵀr; fixed LMM adds λI; residual LMM uses λₖ = c‖rₖ‖. We take the full step wₖ₊₁ = wₖ + p. There is no hidden line search, step clipping, constraint, or retry with increased damping. The parameter sliders restrict only the starting point. Damping can stabilize steps, but does not guarantee finding the best basin.
The small linear systems are solved in double precision. Stop when ‖r‖ ≤ 10⁻¹², ‖∇f‖ ≤ 10⁻¹⁰, or after 200 full steps. A singular or numerically ill-conditioned system, stagnation, or an unsafe numerical evaluation is reported separately. Small gradient alone is not a certificate of a global minimum. The numerical table and CSV retain the actual iterates. Noisy fits and positive-loss local minima are not covered by the zero-residual quadratic convergence theorem.