A1 · Supervised Machine Learning in Depth

Bagging and Random Forests

Bagging (bootstrap aggregating) trains many models on bootstrap samples, random samples of the training data drawn with replacement, and averages their predictions to reduce variance. A random forest bags decision trees and also considers only a random subset of features at each split, which decorrelates the trees. Out-of-bag (OOB) error uses the rows each tree did not see as a built-in validation set.

15 min Advanced Jump to the activity

What you need: The earlier topics of this module, and its prerequisites.

Definition

Why ask a crowd of imperfect experts instead of one?

Bagging (bootstrap aggregating) trains many models on bootstrap samples, random samples of the training data drawn with replacement, and averages their predictions to reduce variance. A random forest bags decision trees and also considers only a random subset of features at each split, which decorrelates the trees. Out-of-bag (OOB) error uses the rows each tree did not see as a built-in validation set.

In simple words

One deep tree memorises its training data and changes wildly if the data changes slightly. Grow hundreds, each on a slightly different sample and each looking at different features, and their individual mistakes cancel out. The vote of the forest is far more stable than any single tree.

Why it matters

Random forests are among the most reliable out-of-the-box models for tabular data: little tuning, no scaling, robust to outliers and with useful feature importances.

Key terms

Bootstrap sample
A sample of n rows drawn with replacement from n rows; about 63 per cent are unique.
Bagging
Bootstrap aggregating: training many models on bootstrap samples and averaging them.
Random forest
Bagged decision trees that also consider a random subset of features at each split.
Out-of-bag (OOB) score
Accuracy measured on the rows each tree did not see, a free validation estimate.
Decorrelation
Making individual trees different so their errors cancel when averaged.
Extra trees
Extremely randomised trees that also pick split thresholds at random.

Explained in detail

Why averaging many trees works

A single deep tree has low bias but high variance: trained on a slightly different sample, it would look very different. Averaging many such trees keeps the low bias while cancelling much of the variance, in the same way that the average guess of a crowd is often better than most individual guesses. Bagging, introduced by Leo Breiman in 1996, trains each tree on a bootstrap sample. Breiman's random forest of 2001 adds a second source of randomness: at each split, only a random subset of features, typically √p for classification and p ÷ 3 for regression, is considered. This stops one strong feature from dominating every tree, making the trees less correlated and the average more accurate.

Useful by-products

Because each bootstrap sample leaves out about 37 per cent of rows, every row is "out of bag" for roughly a third of the trees. Predicting each row using only those trees gives the out-of-bag score, an estimate of test performance without a separate validation set. Random forests also provide feature importance, either impurity-based or, more reliably, permutation-based, and proximity measures that show which rows tend to fall in the same leaves, useful for finding clusters and outliers. They are robust to outliers and irrelevant features, need little tuning and rarely overfit badly as more trees are added; more trees simply make results more stable, at the cost of time.

HyperparameterGuidance
n_estimators200 to 1,000; stop when the OOB score stops improving
max_features"sqrt" for classification; try 0.3 to 0.5 for regression
min_samples_leaf1 to 5 usually; larger smooths noisy data
class_weight"balanced" for imbalanced classes

Random forests are a strong default for tabular data: in many benchmarks they come close to tuned gradient boosting with far less effort. Their limits include large model files, slower prediction than a single tree, and an inability to extrapolate beyond the range of the training targets, since every prediction is an average of training values. They are used widely in finance for credit risk, in ecology for species distribution, in medicine for risk scores and in remote sensing for land-cover classification from satellite images.

How it works

  1. Draw a bootstrap sample of n rows with replacement (about 63 per cent unique rows).
  2. Grow a deep tree, considering a random subset of features (often √p) at each split.
  3. Repeat for many trees.
  4. Predict by majority vote (classification) or averaging (regression).
  5. Estimate error from out-of-bag rows without a separate validation set.

The maths

Variance of an average of B correlated trees
Correlation between trees, reduced by feature sampling
Variance of a single tree
Chance a row is left out of a bootstrap sample

In code

One tree against a bagged ensemble and a random forest, with the out-of-bag estimate.

random_forest.py
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import BaggingClassifier, RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)
models = {
    "single tree": DecisionTreeClassifier(random_state=0),
    "bagged trees": BaggingClassifier(DecisionTreeClassifier(), n_estimators=100, random_state=0),
    "random forest": RandomForestClassifier(n_estimators=200, random_state=0),
}
for name, model in models.items():
    print(f"{name:>13}: {cross_val_score(model, X, y, cv=5).mean():.3f}")

forest = RandomForestClassifier(n_estimators=200, oob_score=True, random_state=0).fit(X, y)
print("Out-of-bag accuracy:", round(forest.oob_score_, 3))

Output

  single tree: 0.917
 bagged trees: 0.958
random forest: 0.960
Out-of-bag accuracy: 0.965

Try it yourself

What does more of each do?

For a random forest, does each change mainly reduce variance, or mainly cost time with little gain?

  • Going from 2,000 trees to 20,000

  • Sampling fewer features per split

  • Going from 10 trees to 200

Common mistakes

  • Expecting a forest to extrapolate beyond the training range. Trees can't.
  • Reading impurity-based importances as exact. They favour high-cardinality features; use permutation importance.
  • Tuning hundreds of parameters. Defaults with enough trees are usually close to the best.

Check yourself

Question 1

1. What does bagging mainly reduce?

Question 2

2. What makes a random forest different from plain bagging?

Question 3

3. About what share of rows is out-of-bag for each tree?

Summary

  • Bagging averages models trained on bootstrap samples.
  • Random forests add feature sampling to decorrelate trees.
  • OOB error is a free validation estimate.

Read it. Now try it yourself.

Open the interactive activity

Finished reading?

Sources: scikit-learn: Random forests · James et al., An Introduction to Statistical Learning