LearnMachine learning
Gradient boosting
Build a strong model from many weak ones, each tree fixing what the previous ones still get wrong.
Press "Add a tree" and watch the curve correct itself. Filled dots train the model; hollow dots test it.
How it works
Small corrections that add up.
Gradient boosting starts with a simple guess, the average. Then it adds tiny trees, each with a single split, and every new tree is trained on the residuals: the errors that remain.
The learning rate η shrinks each correction, so the model improves in many small steps instead of a few big jumps. Smaller steps usually generalise better but need more trees.
Add too many trees and the model starts fitting the noise, just like a curve with too high a degree. That is why boosted models are stopped early, when the test error is lowest. Libraries such as XGBoost and LightGBM win many data competitions this way.