Regularization

An interactive companion to the Konstanz 2026 lecture. Move the sliders.

1. The data-generating process

We fix a true regression function $f:\mathbb{R}\to \mathbb{R}$ and observe noisy samples

with inputs $x_i\in [0,1]$ drawn uniformly. Throughout this page we take $f(x)=\sin (2\pi x)$, the canonical Bishop chapter-1 example, because it is non-polynomial, has interesting curvature, and is impossible to fit exactly with the polynomial function class we’ll use.

Our task is to learn an estimator $\hat{f}$ from a training sample $\mathcal{D}=\{(x_i,y_i){\}}_{i=1}^n$ that approximates $f$ well on new $x$.

The two knobs that control the difficulty of the problem are $n$ and $\sigma$. With $\sigma =0$ and $n$ large, any reasonable estimator will recover $f$ essentially perfectly. With $\sigma$ large and $n$ small, even the true $f$ leaves residuals: the irreducible error ${\sigma }^2$ that no estimator can dodge.

2. Polynomial regression by ordinary least squares

Choose a degree $d\ge 0$ and use the polynomial function class

Stack the training inputs into a Vandermonde design matrix $X\in {\mathbb{R}}^{n\times (d+1)}$ with $X_{ij}=x_i^j$. The OLS estimator minimizes

with the well-known closed form

(In the demos we work in the rescaled coordinate $\tilde{x}=2x-1\in [-1,1]$ so that columns of $X$ stay bounded, but this is a numerical convenience; the math is unchanged.)

Watch what happens as you crank $d$ up at fixed $n$:

• Training MSE decreases monotonically; more parameters can always reduce in-sample error.
• Test MSE traces the famous U-shape: high at small $d$ (the model can’t represent $\sin$), high again at large $d$ (the model fits the noise), with a sweet spot in between.

This is the symptom of the bias–variance tradeoff. To diagnose it properly, we need the decomposition.

3. The bias–variance decomposition

What we ultimately care about is the total error, the expected squared error of our learned $\hat{f}$ averaged over a fresh test input $x\sim p$, the training set $\mathcal{D}$, and the test noise $\epsilon$:

The trick is to fix $x$ first, decompose the inner expectation, and then take the outer one.

Step 1: pointwise decomposition. At a fixed $x$,

Sketch. Add and subtract $\bar{f}(x):={\mathbb{E}}_{\mathcal{D}}[{\hat{f}}_{\mathcal{D}}(x)]$ inside the square, then $f(x)$, expand, and use $\mathbb{E}[\epsilon ]=0$ together with $\epsilon \perp \mathcal{D}$ to kill the cross terms. Three nonnegative pieces remain.

Step 2: integrate over $x$. The outer ${\mathbb{E}}_{x\sim p}$ passes through the equality term-by-term. The noise term is constant in $x$ (homoscedastic), so it survives unchanged:

Reading. Bias is how far the average learner is from the truth, a property of the function class. Variance is how much the learner wobbles across draws of $\mathcal{D}$, a property of how much it is allowed to chase noise. Noise is the floor we cannot beat.

The Monte Carlo. In this demo $p(x)=\mathrm{Uniform}[0,1]$. We draw $K$ independent training sets of size $n$, fit a degree-$d$ polynomial on each, and compute ${\hat{f}}_{\mathcal{D}}(x_g)$ on a uniform grid $\{x_g{\}}_{g=1}^G$. The pointwise quantities ${\text{Bias}}^2(x_g)$ and $\text{Var}(x_g)$ are sample estimates over the $K$ resamples; the scalars ${\mathbb{E}}_x[\cdot ]$ shown below are uniform-grid averages over $g$. Both approximations are simultaneously refined by the same draws.

The top plot shows the ensemble of fits; the second plot is the integrand of $R(\hat{f})$ as a function of $x$, with the three components stacked. The top edge of the stack is the total expected error at each $x$, and the area of each band integrates to its scalar in the legend below.

Things to try:

1. Set $d=0$ (the constant fit). The orange Bias² band dominates and tracks $|f(x)-\bar{f}{|}^2$, visibly larger near the peaks $x\approx 0.25,0.75$ where $|\sin (2\pi x)|$ is maximal. Variance is uniformly tiny; every training set produces nearly the same flat line.
2. Set $d=14$ at small $n$. Variance explodes, especially near $x=0$ and $x=1$ where high-degree polynomials wobble the most given thin boundary support. Bias² stays small everywhere.
3. Slide $\sigma$ up. The gray noise band rises uniformly across $x$; the integrated identity ${\mathbb{E}}_x[{\text{Bias}}^2]+{\mathbb{E}}_x[\text{Var}]+{\sigma }^2$ should still track the empirical test MSE within Monte Carlo error.

This is the picture we are trying to manipulate. Regularization is the lever.

4. Ridge regression: shrinking variance with ${\ell }_2$

The simplest fix for high-variance regimes is to add an ${\ell }_2$ penalty on the weights:

We do not penalize the intercept; let $M=\mathrm{diag}(0,1,1,\dots ,1)$. Setting the gradient to zero gives the closed form

Two limits to keep in mind: $\lambda \to 0$ recovers OLS, and $\lambda \to \infty$ shrinks every non-intercept coefficient to zero (the prediction collapses to $\bar{y}$).

Pin the polynomial degree at the deliberately-too-large $d=12$ and sweep $\lambda$:

• At $\lambda \approx 10^{-6}$ you get the unregularized fit: wiggly, hugging individual data points.
• At $\lambda \approx 10^{-2}$ you typically get something close to the truth, even with $d=12$.
• At $\lambda =10$ you’ve pulled all coefficients toward zero and the curve flattens.

Geometrically, ridge replaces “minimize $\Vert y-Xw{\Vert }^2$” with “minimize $\Vert y-Xw{\Vert }^2$ subject to $\Vert w{\Vert }_2\le t$” for some $t=t(\lambda )$. The constraint is a Euclidean ball, so the solution is shrunk uniformly but rarely exactly zero.

5. Lasso: sparsity from ${\ell }_1$ bonus

This section is extra material — ℓ₁ regularization was not covered in the lecture. Skip ahead to §6 if you only want what was on the slides.

Switching the penalty to ${\ell }_1$,

changes the geometry. The constraint set $\{\Vert w{\Vert }_1\le t\}$ is a polytope with corners on the coordinate axes, so the optimum tends to land on a corner, i.e., with several coefficients exactly zero.

There is no closed form, but the objective splits as $f(w)+g(w)$ with $f$ smooth (the squared loss) and $g$ non-smooth but separable (the ${\ell }_1$). This is the perfect setting for proximal gradient descent, also known here as ISTA:

where ${\mathcal{S}}_{\alpha }$ is the elementwise soft-threshold

(with the intercept entry left alone). The step size $t$ must satisfy $t\le 1/L$ where $L$ is an upper bound on the largest eigenvalue of $X^{\top }X$; we use the Frobenius bound $L=\Vert X{\Vert }_F^2$, which is loose but free.

Push $\lambda$ up and count how many coefficients have collapsed exactly to zero. Lasso isn’t doing rounding-to-zero; those weights are genuinely the optimum of the penalized objective. This is what makes it a feature-selection procedure.

6. Putting it together

Now run the bias–variance Monte Carlo with regularization turned on. At fixed degree $d$, sweeping $\lambda$ is moving along a path through model space that trades bias for variance smoothly.

Recommended exercise: with $d=10$, ridge selected, find the $\lambda$ that minimizes total error. Then switch to lasso and find its optimum. Compare. Finally, switch the penalty off and observe how badly OLS does at the same $d$. The gap is what regularization buys you.

7. The regularization path

So far we’ve moved sliders and watched single-shot views: a fit at one $d$, an integrated decomposition at one $(d,\lambda )$. The natural next move is to sweep $\lambda$ at fixed $d$ and watch how each piece of the total error

traces out a curve. This is the canonical bias–variance tradeoff plot:

Read it left-to-right:

• Small $\lambda$. The penalty is too weak to discipline a degree-$d$ polynomial, so the variance curve dominates. Different training sets give wildly different fits.
• Large $\lambda$. Coefficients are crushed toward zero; the model can’t even represent $f$, so the bias curve dominates.
• Somewhere in between the two curves cross, and the total error has a strict minimum at ${\lambda }^{\ast }$ (red dashed). Below the total curve sits the irreducible noise floor ${\sigma }^2$ (gray dashed). No procedure can ever go below it.

That minimum ${\lambda }^{\ast }$ is the population-optimum hyperparameter, the value we would choose if we knew $f$ and ${\sigma }^2$. We don’t, of course. Cross-validation, AIC, BIC, marginal likelihood under a Gaussian prior, evidence approximation: every model-selection procedure you’ve seen is some way of estimating ${\lambda }^{\ast }$ from data alone, with different sample-size and bias trade-offs of its own. But all of them are picking a point on this same curve.

Things to try:

1. Push $d$ up to 15. The variance curve at small $\lambda$ explodes (high-degree polynomials are very high-capacity); ${\lambda }^{\ast }$ shifts right.
2. Drop $n$ to 15. Variance curve rises across the board; ${\lambda }^{\ast }$ shifts right again. Less data → more regularization needed.
3. Switch to lasso. The bias curve rises in characteristic stair-steps as $\lambda$ crosses thresholds where individual coefficients hit zero.

Nicholas Gao · Konstanz 2026 · all numerics computed live in your browser