Regularization

An interactive companion to the Konstanz 2026 lecture. Each section is paired with a live demonstration; adjust the controls to explore.

1. The data-generating process

We fix a true regression function $f:\mathbb{R}\to \mathbb{R}$ and observe noisy samples

with inputs $x_i\in [0,1]$ drawn uniformly. Throughout this page we take $f(x)=\sin (2\pi x)$, the canonical example from the first chapter of Bishop, chosen because it is non-polynomial, has nontrivial curvature, and cannot be fit exactly by the polynomial function class introduced below.

Our task is to learn an estimator $\hat{f}$ from a training sample $\mathcal{D}=\{(x_i,y_i){\}}_{i=1}^n$ that approximates $f$ well on new $x$.

The two parameters that control the difficulty of the problem are $n$ and $\sigma$. With $\sigma =0$ and $n$ large, any reasonable estimator recovers $f$ essentially perfectly. With $\sigma$ large and $n$ small, even predictions made with the true $f$ leave residuals of variance ${\sigma }^2$: the irreducible error that no estimator can avoid.

2. Polynomial regression by ordinary least squares

Choose a degree $d\ge 0$ and use the polynomial function class

Stack the training inputs into a design matrix $X\in {\mathbb{R}}^{n\times (d+1)}$ with $X_{ij}=x_i^j$. The OLS estimator minimizes

with the well-known closed form

(In the demos we work in the rescaled coordinate $\tilde{x}=2x-1\in [-1,1]$ so that the columns of $X$ remain bounded; this is a numerical convenience and does not affect the analysis.)

As $d$ increases at fixed $n$, two patterns are visible:

• Training MSE decreases monotonically; additional parameters can always reduce the in-sample error.
• Test MSE traces the characteristic U-shape: large at small $d$ (the model cannot represent $\sin$), large again at high $d$ (the model fits the noise), with a minimum at intermediate $d$.

This is the symptom of the bias–variance tradeoff. To analyze it properly we require the decomposition.

3. The bias–variance decomposition

The quantity of interest is the total error, the expected squared error of the learned $\hat{f}$ averaged over a fresh test input $x\sim p$, the training set $\mathcal{D}$, and the test noise $\epsilon$:

A short calculation shows that, at a fixed $x$, this expected error splits into three nonnegative pieces:

Averaging over $x$ extends the same identity to the total error:

Interpretation. Bias measures how far the average learner is from the truth and is a property of the function class. Variance measures how much the learner fluctuates across draws of $\mathcal{D}$ and reflects how strongly it adapts to the noise. The noise term is the irreducible floor that no estimator can fall below.

Estimating the components. Bias and variance are expectations over training sets, so we cannot compute them in closed form. The demo approximates them by simulation: it draws many random training sets, fits a degree-$d$ polynomial to each, and looks at the spread of the resulting predictions. At each $x$, the spread across training sets gives $\text{Var}(x)$, and the gap between the average prediction and $f(x)$ gives ${\text{Bias}}^2(x)$. Averaging these along $x$ yields the scalar values shown in the legend. The MC resamples slider controls the number of training sets — more samples give smoother estimates.

The top plot shows the ensemble of fits; the second plot is the integrand of $R(\hat{f})$ as a function of $x$, with the three components stacked. The top edge of the stack is the total expected error at each $x$, and the area of each band integrates to its scalar in the legend below.

Suggested exercises:

1. Set $d=0$ (the constant fit). The orange Bias² band dominates and tracks $|f(x)-\bar{f}{|}^2$, visibly larger near the peaks $x\approx 0.25,0.75$ where $|\sin (2\pi x)|$ is maximal. Variance is uniformly small; every training set produces nearly the same flat line.
2. Set $d=14$ at small $n$. Variance grows substantially, particularly near $x=0$ and $x=1$, where high-degree polynomials oscillate most strongly given the limited support at the boundaries. Bias² remains small throughout.
3. Increase $\sigma$. The gray noise band rises uniformly across $x$, and the integrated identity ${\mathbb{E}}_x[{\text{Bias}}^2]+{\mathbb{E}}_x[\text{Var}]+{\sigma }^2$ continues to track the empirical test MSE up to Monte Carlo error.

This is the structure we now seek to control. Regularization is the principal tool.

4. Ridge regression: shrinking variance with ${\ell }_2$

A standard remedy in high-variance regimes is to add an ${\ell }_2$ penalty to the weights:

Setting the gradient to zero gives the closed form

Two limiting cases are useful to keep in mind: $\lambda \to 0$ recovers OLS, while $\lambda \to \infty$ shrinks every coefficient to zero.

Fix the polynomial degree at the deliberately large value $d=12$ and vary $\lambda$:

• At $\lambda \approx 10^{-6}$ the fit is essentially unregularized: highly oscillatory, closely tracking individual data points.
• At $\lambda \approx 10^{-2}$ the fit typically lies close to the truth, even at $d=12$.
• At $\lambda =10$ all coefficients have been pulled toward zero and the curve flattens.

As $\lambda$ grows, the quadratic penalty $\lambda \Vert w{\Vert }_2^2$ pulls each coefficient continuously toward zero. The shrinkage is smooth: ridge rarely sets any coefficient exactly to zero.

5. Lasso: sparsity from ${\ell }_1$ bonus

6. Bias–variance with regularization

We now repeat the bias–variance Monte Carlo with a regularization penalty in place. At fixed degree $d$, varying $\lambda$ traces a smooth path through model space that exchanges bias for variance.

Suggested exercises:

1. With $d=10$ and ridge selected, locate the $\lambda$ that minimizes the total error.
2. Switch to lasso and locate its optimum; compare with the ridge result.
3. Disable the penalty (select OLS) and observe the estimator at the same $d$. The gap quantifies the gain from regularization.

7. The bias–variance tradeoff curve

The previous demos have shown single-shot views: one fit at a chosen $d$, one decomposition at a chosen $(d,\lambda )$. We now hold $d$ fixed and sweep $\lambda$, plotting each component of the total error

as a curve in $\lambda$. The result is the canonical bias–variance tradeoff plot:

Reading from left to right:

• Small $\lambda$. The penalty is too weak to constrain a degree-$d$ polynomial, and the variance term dominates. Different training sets produce substantially different fits.
• Large $\lambda$. Coefficients are shrunk toward zero, the model can no longer represent $f$, and the bias term dominates.
• Intermediate $\lambda$. The two curves cross, and the total error attains a strict minimum at ${\lambda }^{\ast }$ (red dashed). The irreducible noise floor ${\sigma }^2$ (gray dashed) lies below the total curve and is unattainable by any estimator.

The minimizer ${\lambda }^{\ast }$ is the population-optimal hyperparameter: the value one would select with full knowledge of $f$ and ${\sigma }^2$. In practice neither is available, so ${\lambda }^{\ast }$ must be estimated from data — typically by a train–validation split or by cross-validation. Both procedures pick a point on this same curve.

Suggested exercises:

1. Increase $d$ to 15. The variance curve at small $\lambda$ rises sharply, since high-degree polynomials carry high capacity, and ${\lambda }^{\ast }$ shifts to the right.
2. Decrease $n$ to 15. The variance curve rises across the entire range and ${\lambda }^{\ast }$ shifts further right: less data calls for stronger regularization.
3. Switch to lasso. The bias curve rises in characteristic stair-steps as $\lambda$ crosses the thresholds at which individual coefficients become exactly zero.

Nicholas Gao · Konstanz 2026 · all numerics computed live in your browser