SGD: stationarity, PL, and variance reduction

The objective has a 1-Lipschitz gradient and a finite infimum. Its noisy gradient has variance σ² at every point. It does not satisfy a global PL inequality: at w = π the gradient vanishes even though f(π) = 2 > f⋆.

The plotted statistic is an average over past iterates, not the gradient at the latest point. An independent uniform choice among w₀,…,w_k−1 has the same expected statistic. A small gradient does not by itself establish global optimality.

Changing T starts a new run with a new fixed step. The step does not shrink as the animation advances. At the end of the run, the bound on the average gradient norm is √[2(f(w₀) − f⋆) + σ²] / T^1/4. This is a worst-case upper bound; the empirical curve need not decay at exactly that rate.

The landscape uses the actual, unwrapped iterates. The chart has no average at k = 0. In logarithmic view, zero and values below 10⁻⁶ appear at the plotting floor; the readout retains their actual values. All horizontal axes are linear.

The oracle is g(w, s) = w + sb. Its variance is b² for every w, so this example satisfies the globally uniform variance assumption. The average objective has L = μ = 1.

For a constant step:

E[f(w_k) − f⋆] = ½(1 − η)^2kw₀² + ηb²/[2(2 − η)] · [1 − (1 − η)^2k].

The limiting gap is ηb²/[2(2 − η)]. The general PL theorem gives an upper bound on a noise neighborhood; this is the exact expectation for this example.

With η_k = 1/(k + 1), the first update uses η₀ = 1. For k ≥ 1, the iterate is the negative average of the sampled offsets:

w_k = −(b/k) Σ_j=0^k−1 s_j,   E[f(w_k) − f⋆] = b²/(2k).

The first update removes dependence on w₀ for this particular quadratic. At k = 0, all methods have gap w₀²/2; the 1/k formula applies only for k ≥ 1. The sampling variance stays b², but the smaller steps average its effect away. All horizontal axes in this demo are linear.

The same s appears in both component gradients. Their linear offsets sb cancel exactly. Moving b changes the raw gradients but leaves the corrected gradients unchanged.

Both estimators have mean w. At w = w̃, SVRG has zero variance, even when the gradient w is nonzero. Its second moment there is w². A stale snapshot can increase variance; it must be refreshed as optimization progresses.

For r > 0, raw variance grows with w and is not globally bounded. This experiment illustrates the control variate directly; the finite-sum analysis uses objective-dependent moment bounds instead of a uniform variance constant.

GD uses η = 1/L_f, where L_f is the smoothness of the average. SGD and SVRG use the selected fraction of 1/L, where L = maxᵢ(‖aᵢ‖² + λ). The two constants can differ substantially.

One GD step costs 120 component gradients; one SGD step costs one. This implementation evaluates both component gradients on every SVRG inner update, including the first. An epoch therefore costs 120 + 2m calls. At the first step, those two evaluations could be saved because the direction equals the stored full gradient.

At the end of an epoch, SVRG averages the pre-update points w₀,…,w_m−1 and starts the next epoch at that average, as in the lecture algorithm. A snapshot can therefore change the current parameter without an additional gradient call. Diamond markers show these events.

The default axis compares equal computational work. Switching to updates compares equal numbers of parameter updates and hides their different costs. The trajectories are individual seeded runs, not expected-value guarantees. Selected parameters are for experimentation; Lecture 12 derives sufficient choices for the theorem.

Objective gaps, contours, and exact estimator variances are diagnostics calculated for visualization; their evaluations are excluded from the algorithm's gradient-call counts. The synthetic dataset is generated once and remains fixed when sampling is reset. No data or libraries are downloaded.