LearnMachine learning
Regularisation (ridge and lasso)
Tame an overfitting model by charging it for large weights, and see lasso switch the weakest features off.
Filled dots train the curve; hollow dots test it. Raise the strength λ and watch the weights shrink. Drag any dot to change the data.
How it works
Simpler models, by paying for complexity.
A flexible model can fit the training data almost perfectly, noise included, and then fail on new data. Regularisation adds a penalty for large weights to the error it minimises, so the model only keeps the complexity that really pays off.
Here the curve is built from twelve features, from gentle (1) to very wiggly (12), and the bars show their weights; the outlines show the weights with no penalty. Ridge charges for the square of each weight and shrinks them all by the same factor. Lasso charges for their size and subtracts the same amount from each, so small weights drop to exactly zero. These simple rules hold because the twelve features here are built to be uncorrelated; with correlated features both penalties still shrink the weights, but less evenly.
The strength λ sets the balance: too little and the model overfits, too much and it underfits. It is usually chosen with cross-validation. Lasso doubles as feature selection, because the features it switches off were not worth their cost.