Forests and Boosting

Why trees still win

Most of the world's valuable data does not look like images or text. It looks like a spreadsheet: one row per customer, loan, patient or shipment, and a few dozen columns of numbers and categories. On that kind of data, the best model is very often not a neural network. It is a few hundred small decision trees added together.

What is it?

An ensemble is a model made of many models whose predictions are combined, by voting, averaging or adding. This lesson is about the two families of tree ensembles that dominate tabular machine learning: random forests, which average many deep trees grown independently, and gradient boosting, which adds up many shallow trees grown one after another, each correcting the last.

The evidence for their strength is unusually consistent. When Tianqi Chen and Carlos Guestrin introduced XGBoost, they counted that 17 of the 29 winning solutions published on Kaggle's blog in 2015 used it. Seven years and a deep learning revolution later, a careful benchmark of 45 tabular datasets by Grinsztajn, Oyallon and Varoquaux found that tree ensembles still outperformed deep networks on medium-sized data (around 10,000 rows), while also being far cheaper to train and tune. Another team re-ran several published deep tabular models on the datasets from their own papers and found XGBoost usually did better.

How does it work?

A single decision tree splits the data with yes-or-no questions on one column at a time (“income ≤ 42k?”), which makes it naturally good at tables: it does not care whether columns are in pounds or percentages, handles skewed distributions and irrelevant features gracefully, and captures sharp thresholds and interactions. Its weakness is instability: change a few rows and the whole tree can change.

Ensembles keep the strengths and fix the weakness. Averaging many trees trained on perturbed data cancels their individual quirks. Adding many small trees, each aimed at what the previous ones got wrong, builds up a complex function one simple correction at a time. Grinsztajn and colleagues traced the neural networks' disadvantage to exactly the properties trees have: networks are biased towards overly smooth functions, are hurt by uninformative columns, and treat columns as interchangeable directions in space when in a table each column means something different.

Why does it matter?

Credit scoring, fraud detection, demand forecasting, insurance pricing, search ranking and clinical risk scores are largely built on tree ensembles. Knowing how they work tells you how to tune them, when they will overfit, and how far to trust the feature importance charts that come with them.

The story is not frozen, which makes it more interesting. In 2025 a team led by Frank Hutter published TabPFN in Nature: a transformer pretrained on millions of synthetic tables that makes predictions for a new table in a single forward pass. On datasets up to 10,000 samples it outperformed tuned tree ensembles, and in 2.8 seconds beat an ensemble of the strongest baselines tuned for 4 hours. Tabular foundation models are the first serious challenger in a decade. You will meet them again at the end of the lesson.

Key takeaways

• An ensemble combines many models; tree ensembles combine many decision trees by averaging (forests) or by adding corrections (boosting).
• On medium-sized tabular data, gradient-boosted trees and random forests still usually beat deep networks, and cost far less to train.
• Pretrained tabular foundation models such as TabPFN are the new challenger, especially on small tables.

The wisdom of crowds

In 1907 Francis Galton collected 787 guesses of the weight of an ox at a country fair. Individual guesses were all over the place. Their median was 1,207 pounds. The ox weighed 1,198.

What is it?

The wisdom of crowds is the observation that the combined judgement of many people is often better than almost any individual in the group. Its mathematical core is older still: in 1785 the Marquis de Condorcet proved what is now called the jury theorem. If each juror is right with probability p greater than one half, and jurors decide independently, then the probability that the majority is right rises towards certainty as the jury grows.

That one word, independently, carries the whole theorem. A crowd of people who all read the same wrong newspaper is no wiser than one reader. Every ensemble method in this lesson is, at heart, a way to get many models that are individually decent and make different mistakes.

How does it work?

Take 25 voters who are each right 60% of the time. The majority is wrong only when 13 or more of them are wrong at once. If their errors are independent, the number who are right follows a binomial distribution, and adding it up gives a majority accuracy of about 85%. With 101 voters it is about 98%. Nothing about any single voter improved; only the combination did.

Now let the voters share a source of error. In the demo, each voter either copies a shared opinion (which is itself right 60% of the time) or thinks independently, and the copying probability is set so that any two voters' correctness has correlation ρ. The accuracies below are computed exactly by summing over how many voters copied and whether the shared opinion was right; nothing is sampled except the grid of dots.

Even a small correlation caps what the crowd can achieve: once most voters lean on the shared opinion, the majority is only as good as that opinion. And below 50% individual accuracy the theorem runs in reverse; a big crowd of voters who are usually wrong is almost always wrong. Ensembles need members that are better than chance and whose errors are as unrelated as possible.

Why does it matter?

This is the design brief for every method that follows. Bagging and random forests manufacture independence by training each tree on a different random view of the data. Boosting takes a different route: instead of hoping errors cancel, it deliberately trains each new model on the errors of the ones before. The same logic explains why teams of language models voting on answers (self-consistency) help most when their reasoning paths genuinely differ.

Key takeaways

• A majority of independent voters who are each right more than half the time becomes almost always right as the crowd grows (Condorcet).
• Correlated errors put a ceiling on the gain: a crowd that shares its mistakes is no wiser than its shared source.
• Good ensembles need members that are better than chance and wrong in different ways.

Bagging

A fully grown decision tree is like an overconfident expert: it fits its training data perfectly and changes its mind completely if you show it slightly different data. Leo Breiman's 1996 insight was that you can turn that instability into an asset.

What is it?

Bagging, short for bootstrap aggregating, trains the same learner many times on bootstrap samples of the training data and averages the results (or takes a majority vote for classes). A bootstrap sample of n rows is drawn from the n training rows with replacement, so some rows appear two or three times and others not at all. Breiman showed that this helps most for exactly the learners that are unstable, with decision trees the classic example.

How does it work?

The chance that a particular row is never picked in n draws is (1 − 1/n)ⁿ, which approaches 1/e ≈ 36.8% for large n. So each bootstrap sample contains about 63% of the distinct rows, some repeated. Each tree sees a slightly different dataset, overfits it in a slightly different way, and those idiosyncrasies are what averaging cancels.

In the language of the bias-variance trade-off, a deep tree has low bias (it can represent almost anything) and high variance (its shape depends heavily on the particular sample). Averaging B trees leaves the bias unchanged and, if the trees were independent, would divide the variance by B. They are not fully independent, since they all come from the same data, which is where the next section picks up.

A single deep tree has a test error well above the noise floor, because it reproduces the noise in its 40 points. The average of a few dozen bootstrap trees is smoother and closer to the true curve, and its test error closes a good part of the gap to the noise floor, even though every tree in it is just as wild as the first. (No model can go below the floor: that is the noise in the test points themselves.) Try a new training sample: the single tree changes shape a lot, the average much less. That stability is the point.

A free test set: out-of-bag error

Since each tree never saw about a third of the rows, those rows can grade it. Score every row using only the trees for which it was out of the bag, and you get the out-of-bag (OOB) error: a nearly unbiased estimate of test error that costs nothing and uses no held-out data. Breiman used it to tune forests without cross-validation, and it is one argument in scikit-learn's RandomForestClassifier(oob_score=True).

Why does it matter?

Bagging is the simplest reliable way to make an unstable model trustworthy, and it parallelises perfectly because the trees never talk to each other. It is also the foundation of the random forest, which remains one of the best models you can train with no tuning at all.

Key takeaways

• Bagging trains a model on many bootstrap samples (drawn with replacement) and averages them.
• It cuts variance and leaves bias alone, so it helps unstable, low-bias learners like deep trees the most.
• Each tree misses about 37% of rows; scoring rows with the trees that never saw them gives the out-of-bag error for free.

Random forests

Bagging has a blind spot. If one feature is very strong, nearly every bootstrap tree splits on it first, and the trees end up as near copies of each other. A crowd of copies is not a crowd.

What is it?

A random forest is bagging with one addition: at every split, the tree may only choose among a random subset of the features, typically about √d of the d features for classification. Tin Kam Ho explored trees built on random feature subspaces in 1995, and Breiman combined the idea with bagging, out-of-bag estimates and permutation importance in his 2001 paper, which became one of the most cited in machine learning.

How does it work?

If B trees each have variance σ² and every pair of them has correlation ρ, the variance of their average is:

The first term does not shrink with more trees. Only making the trees less correlated reduces it.

As B grows, the second term vanishes and the first is left. With ρ = 0.6 you can never do better than 60% of one tree's variance, no matter how many trees you grow. Random feature subsets attack ρ directly: when the dominant feature is not on offer at a split, the tree must find another way to split, and different trees find different ways. Each tree gets a little worse (slightly higher bias), but the average gets much better.

What you tune, and what you do not

Random forests are famous for working well out of the box. The main settings are the number of features per split (lower means more decorrelated but individually weaker trees), the minimum leaf size or depth (deeper means less bias, more variance per tree), and the number of trees. The last one is not really a tuning parameter: more trees never make a forest overfit, they only cost time. Stop adding them once the out-of-bag error has flattened.

Why does it matter?

A random forest is often the right first model on a new table: hard to break, parallel, comes with a built-in validation estimate, and gives a score you can trust before you spend a day tuning. It is also a common workhorse in science, from genomics to remote sensing, where datasets are modest and robustness matters more than the last fraction of a percent.

Key takeaways

• A random forest is bagging plus a random subset of features at every split.
• Averaging cannot push variance below ρσ²; feature subsampling lowers the correlation ρ between trees.
• More trees never overfit a forest; depth and leaf size control how much each tree overfits.

AdaBoost

In 1988 Michael Kearns asked a deceptively simple question: if you have a learner that is only slightly better than guessing, can you always turn it into one that is almost always right? Robert Schapire proved the answer was yes. The practical algorithm that came out of it, AdaBoost, won its authors the Gödel Prize.

What is it?

Boosting builds an ensemble sequentially. Each new model is trained to do well on the examples the ensemble so far handles badly. AdaBoost (adaptive boosting), introduced by Yoav Freund and Robert Schapire, does this by keeping a weight on every training example and giving the misclassified ones more weight each round. Its classic weak learner is a decision stump, a tree with a single split.

How does it work?

Start with equal weights 1/n on all n points. Then, for each round t:

1. Fit a stump that minimises the weighted error on the training data.
2. Measure its weighted error ε (the total weight of the points it gets wrong).
3. Give it a vote α = ½ ln((1 − ε) / ε). A stump with ε = 0.1 gets α ≈ 1.10; one at ε = 0.45 gets α ≈ 0.10; one at 0.5 would get zero.
4. Multiply the weight of each misclassified point by e^α and each correct one by e^−α, then renormalise so the weights sum to 1.

The final classifier is the sign of the weighted vote Σ αₜhₜ(x). A neat consequence of step 4 is that after reweighting, the stump just added has a weighted error of exactly 50%: the next stump is forced to find something new.

One stump is a crude straight cut. After a few rounds the weight has piled onto the tips of the moons, where the cut is wrong, and the stumps that follow carve those regions out. Around twenty rounds in, a boundary made only of vertical and horizontal cuts traces the curved gap between the classes. Notice also how α changes: later stumps, fighting over the hardest points, have errors closer to 50% and get smaller votes.

What AdaBoost is really optimising

For years AdaBoost's success was a bit mysterious. In 2000, Friedman, Hastie and Tibshirani showed that it is a stagewise way of fitting an additive model F(x) = Σ αₜhₜ(x) that minimises the exponential loss e^−yF(x), with labels y = ±1 (Friedman et al., 2000). The exponential explains both its strength and its weakness: points far on the wrong side of the boundary get exponentially large weights, which focuses effort, but also makes AdaBoost sensitive to mislabelled points and outliers. The recognition that boosting is loss minimisation led directly to the next section.

Why does it matter?

AdaBoost proved in practice what theory had promised: weak models can be combined into a strong one. It powered the Viola-Jones face detector in early 2000s cameras and set the template, reweight or refit on what is still wrong, that every modern boosting library follows.

Key takeaways

• Boosting builds models in sequence, each one focused on what the ensemble so far gets wrong.
• AdaBoost upweights misclassified points and gives each weak learner a vote α = ½ ln((1 − ε)/ε).
• It minimises exponential loss, which makes it sharp but sensitive to label noise.

Gradient boosting

Imagine fitting a curve by hand. You draw a rough first guess, look at where it misses, and sketch a small correction for those misses. Then you look at what is still wrong and sketch another. Gradient boosting is that process, made precise.

What is it?

Gradient boosting, formulated by Jerome Friedman, builds a model F as a sum of small trees, where each new tree is fitted to the residuals (more generally, the negative gradient of the loss) of the model so far. Where AdaBoost reweights points, gradient boosting changes the target: each tree tries to predict what the ensemble still gets wrong.

How does it work?

For squared error, the recipe is short enough to do on paper:

1. Start with a constant: F₀(x) = the mean of y.
2. Compute residuals rᵢ = yᵢ − F(xᵢ) for every training point.
3. Fit a small regression tree h to the residuals.
4. Update F(x) ← F(x) + η · h(x), where η is the learning rate, and repeat from step 2.

Why “gradient”? For the loss L = ½(y − F)², the derivative with respect to the prediction F is −(y − F): the negative gradient is exactly the residual. So each tree approximates the direction in which changing the predictions would reduce the loss fastest, and adding η times that tree is a step of gradient descent in function space. Swap in another loss and the recipe still works: for classification with log loss the “residual” is y − p, the label minus the predicted probability.

Each round adds a shrunken tree fitted to the negative gradient of the loss at the current predictions.

With learning rate 1 and stumps, the first few trees make big, visible jumps and the staircase quickly takes the curve's shape. Each tree's purple step sits exactly where the residuals are largest. Keep going and the training error keeps falling towards zero, while the test error bottoms out and then creeps back up: later trees are fitting noise. With stumps the rise is gentle; switch to depth 3 and training error hits zero within a few dozen trees while test error climbs by half again. The dashed marker is the best number of trees for this data. Compare the best test error at learning rate 1 and at 0.1: the smaller steps need many more trees but end slightly lower.

Shrinkage and early stopping

Friedman found empirically that small learning rates (η around 0.1 or below) give better models than taking the full step, at the cost of more trees. Shrinkage means no single tree can commit too hard to the quirks of the data; the model approaches the target cautiously, and the region around the best number of trees becomes wider and lower. The partner technique is early stopping: hold out a validation set, track its error after every tree, and keep the model from the round where it was lowest. Every modern library does this with one argument.

Two more regularisers matter in practice. Row subsampling fits each tree on a random fraction of the rows (Friedman's stochastic gradient boosting), which adds a little of bagging's variance reduction and speeds things up. Shallow trees, typically depth 3 to 8, limit how complex each correction can be; the depth sets the order of interactions the model can capture.

Why does it matter?

Gradient boosting is the engine behind XGBoost, LightGBM and CatBoost, and so behind a large share of production models on tabular data. The function-space view also makes it general: any differentiable loss (squared error, log loss, ranking losses, quantile loss for prediction intervals, Poisson loss for counts) plugs into the same loop.

Key takeaways

• Gradient boosting adds small trees one at a time, each fitted to the residuals (negative gradient) of the current model.
• It is gradient descent in the space of functions, so any differentiable loss works.
• A small learning rate plus early stopping on validation data is the standard recipe against overfitting.

XGBoost and friends

Friedman's algorithm was published in 2001. What made it dominant a decade later was engineering: three open-source libraries that made boosting faster, better regularised and easier to use on messy real data.

What is it?

XGBoost (2014, paper 2016), LightGBM (Microsoft, 2017) and CatBoost (Yandex, 2017, paper 2018) are gradient-boosted tree libraries. They share the core loop you just ran and differ in how they grow trees, find splits and handle categories.

How does it work?

In practice all three reach similar accuracy on most tables once tuned; the differences are speed, memory and how much preprocessing you need. A reasonable starting configuration for any of them: learning rate 0.05, up to a few thousand rounds with early stopping on a validation set (patience around 50 rounds), trees of depth 6 or about 31 leaves, row and column subsampling around 0.8. Then tune depth or leaves, the minimum samples per leaf, and regularisation.

When to reach for something else

Tree ensembles are the default for tables, not a law of nature. Reach for something else when:

• The data is not really tabular. Images, audio and free text have spatial or sequential structure that convolutional networks and transformers exploit and trees cannot. A common pattern is to turn text or images into embeddings with a neural network and feed those, plus the ordinary columns, to a boosted model.
• You need smooth extrapolation. Trees predict constants in each leaf, so outside the range of the training data their prediction is flat. A demand model trained on prices up to £20 will say the same thing at £30 as at £20.
• The table is small. With hundreds to a few thousand rows, pretrained tabular foundation models like TabPFN have posted strong wins, with no hyperparameter tuning. They are newer, heavier at prediction time and less battle-tested, so compare them against a boosted baseline with cross-validation.
• The data is huge and homogeneous, such as billions of interaction logs in a recommender. Deep models with learned embeddings often win there, and many large systems combine both.

Why does it matter?

Choosing a model family is often the highest-leverage decision in a tabular project, and the wrong default (a big network on 20,000 rows, or a single tree for a production risk score) wastes weeks. Knowing what each library changes also tells you what its knobs do: leaf-wise LightGBM overfits small data faster unless you cap the number of leaves; CatBoost saves you from hand-encoding high-cardinality categories.

Key takeaways

• XGBoost, LightGBM and CatBoost are gradient boosting with regularised objectives, fast histogram or second-order split finding, and careful handling of missing values and categories.
• Start with a small learning rate, many rounds and early stopping; then tune depth or leaves and regularisation.
• Prefer other models for images and text, for extrapolation, or try tabular foundation models on small tables.

Feature importance

Every forest and boosting library will happily print a bar chart ranking your features. Stakeholders love that chart. It can also be quietly wrong, in ways that are easy to demonstrate.

What is it?

Feature importance scores estimate how much a model relies on each input. Three measures are common for tree ensembles:

• Impurity importance (mean decrease in impurity, MDI): add up how much each feature's splits reduced impurity or squared error, across all trees. Free to compute, and the default in many libraries.
• Permutation importance: shuffle one column in held-out data and measure how much the model's score drops. Introduced for forests by Breiman.
• SHAP values: split each individual prediction into additive contributions from each feature using Shapley values from game theory. Lundberg and colleagues' TreeSHAP computes them exactly for tree ensembles in polynomial time, giving both per-prediction explanations and global summaries.

How does it work?

In the demo below, house prices depend only on size and location. Two further columns are pure noise: a random listing ID, and a random yes/no flag. A real 40-tree random forest is trained on 300 rows, and both importance measures are computed from it. You can then add a sixth column, “rooms”, which is almost a copy of size.

The random listing ID picks up a noticeable share of impurity importance: deep trees find chance splits on a column with a distinct value in every row, and every such split counts. Its permutation importance on held-out data is close to zero, which is the truth. This bias of impurity importance towards high-cardinality features was documented by Strobl and colleagues. The binary flag, with only one possible split, barely registers under either measure.

Adding “rooms” exposes the second trap. The forest now splits on size or rooms more or less at random, so impurity importance is shared between them, and shuffling either one alone hurts little because its twin still carries the information. Neither measure is lying about the model; both are answering “how much does the model need this exact column, given the others?”, which is not the same as “how much does this quantity matter?”

Why does it matter?

Importance charts drive real decisions: which data to keep collecting, what to tell a regulator, whether a model is using a forbidden attribute through a proxy. Prefer permutation importance on held-out data or SHAP over raw impurity scores, group correlated features before reading them, and treat the output as a description of the model, not of the world.

Key takeaways

• Impurity importance is free but biased towards features with many distinct values; permutation importance on held-out data avoids that.
• Correlated features share importance under every measure; group or remove near-duplicates before interpreting.
• TreeSHAP gives exact per-prediction attributions for tree ensembles, but all importances describe the model, not causes in the world.

Check your understanding

Seven decisions you might face on a real tabular project. Each one asks you to apply an idea from the lesson.

A retailer has 80,000 rows of customer data with 40 columns (spend, visits, region, device, tenure...) and wants to predict churn. A colleague proposes a large neural network because “deep learning is state of the art”. What is the sensible first model?

Build every ensemble from this lesson yourself, on six datasets, in the Boosting lab. For the theory in depth, chapters 8, 10, 15 and 16 of The Elements of Statistical Learning cover bagging, boosting, random forests and ensembles, and the book is free online.

References

The papers and books behind the algorithms, numbers and claims in this lesson.

References

1. [1]

   XGBoost: A Scalable Tree Boosting System(opens in a new tab)

   Tianqi Chen & Carlos Guestrin, 2016

   KDD 2016. Regularised objective, second-order split scoring, sparsity-aware splits and systems engineering; reports XGBoost in 17 of 29 Kaggle winning solutions published in 2015.

2. [2]

   Why do tree-based models still outperform deep learning on typical tabular data?(opens in a new tab)

   Léo Grinsztajn, Edouard Oyallon & Gaël Varoquaux, 2022

   NeurIPS 2022 Datasets and Benchmarks. A benchmark of 45 medium-sized tabular datasets where tree ensembles beat deep networks, with an analysis of why.

3. [3]

   Tabular data: Deep learning is not all you need(opens in a new tab)

   Ravid Shwartz-Ziv & Amitai Armon, 2022

   Information Fusion 81. Re-tests deep tabular models on datasets from their own papers and finds XGBoost usually does better.

4. [4]

   Accurate predictions on small data with a tabular foundation model(opens in a new tab)

   Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister & Frank Hutter, 2025

   Nature 637. TabPFN, a transformer pretrained on synthetic datasets that predicts on a new table in one forward pass; strongest on datasets up to 10,000 samples.

5. [5]

   Vox populi(opens in a new tab)

   Francis Galton, 1907

   Nature 75. The median of 787 guesses of an ox’s dressed weight at a country fair was 1,207 lb; the true weight was 1,198 lb.

6. [6]

   Essai sur l’application de l’analyse à la probabilité des décisions rendues à la pluralité des voix

   Nicolas de Condorcet, 1785

   The jury theorem: a majority of independent voters who are each right more often than not becomes almost certainly right as the group grows.

7. [7]

   Bagging predictors(opens in a new tab)

   Leo Breiman, 1996

   Machine Learning 24. Bootstrap aggregating: average models trained on bootstrap resamples to reduce the variance of unstable learners.

8. [8]

   Random decision forests(opens in a new tab)

   Tin Kam Ho, 1995

   ICDAR 1995. Trees built in random subspaces of the features, an ancestor of random forests.

9. [9]

   Random forests(opens in a new tab)

   Leo Breiman, 2001

   Machine Learning 45. Bagged trees with random feature selection at each split, out-of-bag estimates and permutation importance.

10. [10]

    The strength of weak learnability(opens in a new tab)

    Robert E. Schapire, 1990

    Machine Learning 5. Proves that any learner slightly better than chance can be boosted into an arbitrarily accurate one.

11. [11]

    A decision-theoretic generalization of on-line learning and an application to boosting(opens in a new tab)

    Yoav Freund & Robert E. Schapire, 1997

    Journal of Computer and System Sciences 55. Introduces AdaBoost.

12. [12]

    Additive logistic regression: a statistical view of boosting(opens in a new tab)

    Jerome Friedman, Trevor Hastie & Robert Tibshirani, 2000

    Annals of Statistics 28. Shows AdaBoost is stagewise fitting of an additive model under exponential loss.

13. [13]

    Greedy function approximation: A gradient boosting machine(opens in a new tab)

    Jerome H. Friedman, 2001

    Annals of Statistics 29. Gradient boosting as gradient descent in function space, with shrinkage and tree-based leaf updates.

14. [14]

    Stochastic gradient boosting(opens in a new tab)

    Jerome H. Friedman, 2002

    Computational Statistics & Data Analysis 38. Fitting each tree on a random subsample improves accuracy and speed.

15. [15]

    LightGBM: A Highly Efficient Gradient Boosting Decision Tree(opens in a new tab)

    Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye & Tie-Yan Liu, 2017

    NeurIPS 2017. Histogram-based splits, leaf-wise growth, gradient-based one-side sampling and exclusive feature bundling.

16. [16]

    CatBoost: unbiased boosting with categorical features(opens in a new tab)

    Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush & Andrey Gulin, 2018

    NeurIPS 2018. Ordered boosting and ordered target statistics to avoid target leakage, especially with categorical features.

17. [17]

    From local explanations to global understanding with explainable AI for trees(opens in a new tab)

    Scott M. Lundberg, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan M. Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal & Su-In Lee, 2020

    Nature Machine Intelligence 2. TreeSHAP: exact Shapley value explanations for tree ensembles in polynomial time.

18. [18]

    Bias in random forest variable importance measures: Illustrations, sources and a solution(opens in a new tab)

    Carolin Strobl, Anne-Laure Boulesteix, Achim Zeileis & Torsten Hothorn, 2007

    BMC Bioinformatics 8. Impurity-based importance favours features with many possible split points.

19. [19]

    The Elements of Statistical Learning (2nd edition)(opens in a new tab)

    Trevor Hastie, Robert Tibshirani & Jerome Friedman, 2009

    Springer. Chapters 8, 10, 15 and 16 cover bagging, boosting, random forests and ensemble learning; free online.