Boosting Lab

About

One decision tree is easy to read but fragile: move a few points and it grows a different shape. Ensembles fix that by combining many trees. There are two opposite ways to do it, and this lab lets you watch both being built one tree at a time.

Averaging (bagging and random forests) grows many deep, independent trees on shuffled copies of the data and averages them. Each tree overfits in its own way; the average cancels much of that noise out.

Boosting (AdaBoost and gradient boosting) grows small trees in sequence, each one aimed at what the ensemble so far still gets wrong. The trees are weak on their own; the sum is strong.

Reading the workspace

• The big map is the ensemble's probability of class B in viridis, with its decision boundary drawn in ink.
• Point size shows what boosting is paying attention to: AdaBoost's weights, or the size of each gradient boosting residual.
• The strip shows the last few trees on their own, so you can compare one tree with the crowd.
• The curves show train and test error for every ensemble size. The faint part is the future; tap it to jump.

How it works

Every tree here is CART: at each node it tries every threshold on every allowed feature and keeps the split that most reduces the weighted squared error of the targets. For 0/1 class labels, that is exactly the Gini criterion, and a leaf predicts the fraction of class B it contains.

Averaging

For bagging (bootstrap aggregating), each tree trains on n points drawn with replacement from the n training points. About 63% of the distinct points land in each sample, some several times; the other 37% are out of bag for that tree. The ensemble predicts the average of the trees' probabilities.

A random forest adds one twist: each split may only consider a random subset of features (max_features). This forces trees to differ even when one feature dominates, which makes their errors less correlated and the average more useful.

Boosting

AdaBoost keeps a weight on every training point. Each round fits a weak tree to the weighted data, measures its weighted error ε, gives it a vote α = ½ ln((1 − ε) / ε), and multiplies the weight of every misclassified point by e^α (and every correct one by e^−α) before renormalising.

Gradient boosting starts from a constant and repeatedly fits a small regression tree to the residuals, the negative gradient of the loss. For classification with log loss the residual is y − p, and each leaf value is a Newton step. The new tree is added after shrinking it by the learning rate.

Bagging and forests

Pick Tree with depth 14 on Noisy blobs: the single tree carves little islands around every flipped label and its test accuracy is well below its training accuracy. Now pick Bagging and press play. Each new tree is just as wild, as the strip shows, but the average smooths the islands away, and test accuracy rises with the number of trees before levelling off.

That curve flattens and stays flat. Adding more trees to a bagged ensemble or random forest does not cause overfitting; it only makes the average more stable. The trees themselves can still overfit, which is what depth limits and minimum leaf sizes control.

Out-of-bag error

Every point is left out of roughly a third of the bootstrap samples. Score each point using only those trees and you get an honest estimate of test error for free, without holding any data back. Watch the grey out-of-bag curve track the test curve once there are enough trees.

Why only two features matters here

With two features, a forest can only choose between “one random feature per split” and “both”. The effect is mild. On real tables with dozens or hundreds of columns, trying about √d features per split (scikit-learn's default for classification) decorrelates the trees far more.

Boosting

Switch to Regression and Gradient with depth 1 and learning rate 1. The first tree is a single step. The red sticks are residuals; the next tree fits them, and the staircase gets one more step. Point size shows how big each residual still is. Drop the learning rate to 0.1 and each tree only moves the fit a tenth of the way, so the staircase creeps towards the curve and you can watch it happen.

Now go back to Classification, pick AdaBoost and step through the first rounds. After each weak vote, the points it got wrong grow. The next stump is forced to care about them. The α and weighted error of each stump are printed under it in the strip.

Boosting can overfit

Unlike averaging, every boosting round reduces training error, so eventually it starts fitting noise. The test curve turns back up (or accuracy starts slipping) after the marked best point. Real libraries stop early: they watch a validation set and keep the number of rounds where it was best. Shrinkage, row subsampling and shallow trees all slow the overfitting down.

Try this

1. 01One tree versus manyNoisy blobs, Tree at depth 14, then Bagging with 60 trees. Compare test accuracy and look at how the islands vanish.
2. 02Stumps cannot do XORXOR with AdaBoost at depth 1: stuck near 50%. Set depth to 2 and step once or twice.
3. 03Watch the weightsMoons, AdaBoost, depth 1. Step slowly. Which points grow first? Where does the next stump put its split?
4. 04Staircase regressionRegression, Wave, Gradient, depth 1, learning rate 1. Step 10 times, then try learning rate 0.1 and 150 rounds.
5. 05Find the overfit pointRegression, Chirp, noise 0.6, Gradient with depth 3 and learning rate 0.5. Where does test MSE bottom out?
6. 06Subsampling as regulariserSame setup, set row subsample to 50%. Does the best test MSE improve? Does training error fall more slowly?
7. 07Out-of-bag honestyCircles, Forest with 100 trees. Compare the out-of-bag curve with the test curve as trees are added.
8. 08Break a boundaryMoons with Forest. Tap a cluster of class B points deep inside class A. How many do you need before the forest believes them?