Definition
Why ask a crowd of imperfect experts instead of one?
Bagging (bootstrap aggregating) trains many models on bootstrap samples, random samples of the training data drawn with replacement, and averages their predictions to reduce variance. A random forest bags decision trees and also considers only a random subset of features at each split, which decorrelates the trees. Out-of-bag (OOB) error uses the rows each tree did not see as a built-in validation set.
In simple words
One deep tree memorises its training data and changes wildly if the data changes slightly. Grow hundreds, each on a slightly different sample and each looking at different features, and their individual mistakes cancel out. The vote of the forest is far more stable than any single tree.
Why it matters
Random forests are among the most reliable out-of-the-box models for tabular data: little tuning, no scaling, robust to outliers and with useful feature importances.
Key terms
- Bootstrap sample
- A sample of n rows drawn with replacement from n rows; about 63 per cent are unique.
- Bagging
- Bootstrap aggregating: training many models on bootstrap samples and averaging them.
- Random forest
- Bagged decision trees that also consider a random subset of features at each split.
- Out-of-bag (OOB) score
- Accuracy measured on the rows each tree did not see, a free validation estimate.
- Decorrelation
- Making individual trees different so their errors cancel when averaged.
- Extra trees
- Extremely randomised trees that also pick split thresholds at random.
Explained in detail
Why averaging many trees works
A single deep tree has low bias but high variance: trained on a slightly different sample, it would look very different. Averaging many such trees keeps the low bias while cancelling much of the variance, in the same way that the average guess of a crowd is often better than most individual guesses. Bagging, introduced by Leo Breiman in 1996, trains each tree on a bootstrap sample. Breiman's random forest of 2001 adds a second source of randomness: at each split, only a random subset of features, typically √p for classification and p ÷ 3 for regression, is considered. This stops one strong feature from dominating every tree, making the trees less correlated and the average more accurate.
Useful by-products
Because each bootstrap sample leaves out about 37 per cent of rows, every row is "out of bag" for roughly a third of the trees. Predicting each row using only those trees gives the out-of-bag score, an estimate of test performance without a separate validation set. Random forests also provide feature importance, either impurity-based or, more reliably, permutation-based, and proximity measures that show which rows tend to fall in the same leaves, useful for finding clusters and outliers. They are robust to outliers and irrelevant features, need little tuning and rarely overfit badly as more trees are added; more trees simply make results more stable, at the cost of time.
| Hyperparameter | Guidance |
|---|---|
| n_estimators | 200 to 1,000; stop when the OOB score stops improving |
| max_features | "sqrt" for classification; try 0.3 to 0.5 for regression |
| min_samples_leaf | 1 to 5 usually; larger smooths noisy data |
| class_weight | "balanced" for imbalanced classes |
Random forests are a strong default for tabular data: in many benchmarks they come close to tuned gradient boosting with far less effort. Their limits include large model files, slower prediction than a single tree, and an inability to extrapolate beyond the range of the training targets, since every prediction is an average of training values. They are used widely in finance for credit risk, in ecology for species distribution, in medicine for risk scores and in remote sensing for land-cover classification from satellite images.
How it works
- Draw a bootstrap sample of n rows with replacement (about 63 per cent unique rows).
- Grow a deep tree, considering a random subset of features (often √p) at each split.
- Repeat for many trees.
- Predict by majority vote (classification) or averaging (regression).
- Estimate error from out-of-bag rows without a separate validation set.
The maths
- Correlation between trees, reduced by feature sampling
- Variance of a single tree
In code
One tree against a bagged ensemble and a random forest, with the out-of-bag estimate.
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import BaggingClassifier, RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier
X, y = load_breast_cancer(return_X_y=True)
models = {
"single tree": DecisionTreeClassifier(random_state=0),
"bagged trees": BaggingClassifier(DecisionTreeClassifier(), n_estimators=100, random_state=0),
"random forest": RandomForestClassifier(n_estimators=200, random_state=0),
}
for name, model in models.items():
print(f"{name:>13}: {cross_val_score(model, X, y, cv=5).mean():.3f}")
forest = RandomForestClassifier(n_estimators=200, oob_score=True, random_state=0).fit(X, y)
print("Out-of-bag accuracy:", round(forest.oob_score_, 3))Output
single tree: 0.917 bagged trees: 0.958 random forest: 0.960 Out-of-bag accuracy: 0.965
Try it yourself
What does more of each do?
For a random forest, does each change mainly reduce variance, or mainly cost time with little gain?
Going from 2,000 trees to 20,000
Sampling fewer features per split
Going from 10 trees to 200
Common mistakes
- Expecting a forest to extrapolate beyond the training range. Trees can't.
- Reading impurity-based importances as exact. They favour high-cardinality features; use permutation importance.
- Tuning hundreds of parameters. Defaults with enough trees are usually close to the best.
Check yourself
Summary
- Bagging averages models trained on bootstrap samples.
- Random forests add feature sampling to decorrelate trees.
- OOB error is a free validation estimate.
Read it. Now try it yourself.
Open the interactive activityFinished reading?
Sources: scikit-learn: Random forests · James et al., An Introduction to Statistical Learning