Definition
Ask one person to guess the number of sweets in a jar and they may be far off. Ask a hundred people and average their guesses, and the answer is often remarkably close. A random forest applies this wisdom of crowds to decision trees.
An ensemble combines the predictions of several models to get a better one. A random forest is an ensemble of decision trees in which each tree is trained on a bootstrap sample of the rows (bagging) and, at every split, considers only a random subset of the features. For classification the trees vote, and for regression their predictions are averaged. The randomness makes the trees different from each other, so their individual errors partly cancel. Rows left out of a tree's bootstrap sample give an out-of-bag estimate of accuracy.
In simple words
One tree is like one doctor with strong opinions shaped by the few patients they saw. A forest is like a panel of a hundred doctors, each trained on different patients and each looking at different test results; their majority opinion is usually more reliable.
Overview
- An ensemble combines several models to get better predictions than any one of them.
- A random forest averages many decision trees trained on bootstrap samples with random feature subsets at each split.
- Averaging decorrelated trees reduces variance and overfitting.
- Rows left out of each bootstrap sample give a free out-of-bag accuracy estimate.
- Forests need little tuning and no scaling, but cannot extrapolate beyond the training targets.
- Leo Breiman introduced random forests in 2001.
Why it matters
Random forests are among the most reliable out-of-the-box methods for tabular data: accurate, robust to outliers and irrelevant features, and needing little tuning. They are used in banking, medicine, agriculture, ecology and remote sensing.
Key terms
- Ensemble
- A group of models whose predictions are combined.
- Bootstrap sample
- Rows drawn at random with replacement, the same number as the original data.
- Bagging
- Bootstrap aggregating: training models on bootstrap samples and averaging them.
- Feature subsampling
- Considering only a random subset of features at each split (max_features).
- Out-of-bag (OOB) score
- Accuracy on rows not used to build each tree, a built-in validation estimate.
- Voting
- Combining classifiers by majority (hard) or by averaging probabilities (soft).
Types
There are several types of ensemble. They differ in how the models are trained and combined.
1. Voting and averaging ensembles
Combining different models by majority vote (hard) or by averaging probabilities (soft).
- How it works: VotingClassifier trains each model separately and combines their outputs; soft voting usually works better.
- Example: Averaging logistic regression, a random forest and k-NN for loan default prediction.
- When to use it: Use it for a quick gain from models that make different kinds of mistakes.
2. Bagging
Training the same model on many bootstrap samples and averaging.
- How it works: Each bootstrap sample draws n rows with replacement, so about 63% of rows appear in it; averaging reduces variance.
- Example: BaggingClassifier with 100 decision trees.
- When to use it: Use it to stabilise high-variance models such as deep trees. Out-of-bag rows give a free accuracy check.
3. Random forest and extra trees
Bagged trees with random feature selection at each split; extra trees also choose split thresholds at random.
- How it works: Feature randomness makes the trees more different from each other, so averaging cancels more of their errors.
- Example: A 300-tree forest scoring 0.87 test accuracy against 0.83 for a single tree.
- When to use it: Use them as a strong, low-effort default for tabular data.
4. Boosting
Training models in sequence, each focusing on the errors of the previous ones.
- How it works: AdaBoost reweights misclassified examples; gradient boosting fits each new tree to the residuals.
- Example: XGBoost and LightGBM models winning tabular-data competitions.
- When to use it: Use it for the highest accuracy on tabular data, with careful tuning and early stopping.
5. Stacking
Training a meta-model that learns how best to combine base models' predictions.
- How it works: Base models produce out-of-fold predictions, which become the features for the meta-model, often a simple linear model.
- Example: Combining gradient boosting, k-NN and a linear model to predict crop yield.
- When to use it: Use it when squeezing out the last bit of accuracy, as in competitions.
Explained in detail
Why many trees beat one
A deep decision tree fits its training data closely, so it has low bias but high variance: train it on a slightly different sample and the splits, and the predictions, change a lot. Averaging many such trees keeps the low bias and cancels much of the variance. But averaging only helps if the trees make different mistakes. Bagging creates difference by giving each tree a different bootstrap sample, in which about 63 per cent of the original rows appear, some several times. Random feature subsampling adds more difference: if one feature, such as income, is very strong, plain bagged trees would all split on it first and look alike, while a forest that considers, say, 4 random features out of 16 at each split forces trees to explore other patterns too.
Ensemble methods compared
| Method | How models are combined | Main benefit |
|---|---|---|
| Voting | Different model types vote or average probabilities | Simple gains from diverse models |
| Bagging | Same model on bootstrap samples, averaged | Reduces variance |
| Random forest | Bagged trees plus random features at each split | Lower variance through decorrelated trees |
| Boosting | Models built in sequence, each fixing earlier errors | Reduces bias; often the most accurate |
| Stacking | A meta-model learns how to combine base models | Can combine strengths of very different models |
Using a random forest well
Key settings are n_estimators, the number of trees (200 to 500 is usually plenty; more trees never cause overfitting but cost time); max_features, the number of features tried at each split (the square root of the number of features is the default for classification); and min_samples_leaf or max_depth, which limit how detailed each tree becomes. Setting oob_score=True gives a free estimate of accuracy. Feature importance shows which inputs the forest relies on, but impurity-based importance favours features with many distinct values, so permutation importance on validation data is more trustworthy. Forests cannot extrapolate: for a house-price forest trained on homes up to ₹2 crore, a mansion will still be predicted at around ₹2 crore. In the code below, a single tree scores 0.83 on test data while a 300-tree forest scores 0.87, and the out-of-bag estimate of 0.87 matches the test score without touching the test set.
How it works
- Start with a single decision tree as a baseline.
- Train a random forest with 300 trees and oob_score=True.
- Tune max_features and min_samples_leaf with cross-validation if needed.
- Check permutation importance on validation data.
- Compare with gradient boosting and logistic regression before choosing.
The maths
- Number of trees
Five trees classify a loan applicant: three predict "repay" and two predict "default", so the forest's hard vote is "repay". Their probabilities of default are 0.30, 0.40, 0.45, 0.70 and 0.80; the average is (0.30 + 0.40 + 0.45 + 0.70 + 0.80) ÷ 5 = 0.53, so soft voting predicts "default" at a 0.5 threshold. This shows that hard and soft voting can disagree when the minority trees are very confident, and soft voting usually performs better.
In code
import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
import warnings
X, y = make_classification(n_samples=1200, n_features=10, n_informative=4, n_redundant=2, flip_y=0.08, random_state=1)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0)
tree = DecisionTreeClassifier(random_state=0).fit(X_tr, y_tr)
print(f"Single tree train={tree.score(X_tr, y_tr):.2f} test={tree.score(X_te, y_te):.2f}")
for n in [10, 100, 300]:
with warnings.catch_warnings():
# With only 10 trees a few rows are never left out of a sample, so scikit-learn warns.
# The OOB score still uses every other row; more trees make it more reliable.
warnings.simplefilter("ignore", UserWarning)
forest = RandomForestClassifier(n_estimators=n, oob_score=True, random_state=0).fit(X_tr, y_tr)
print(f"Forest {n:>3} trees train={forest.score(X_tr, y_tr):.2f} test={forest.score(X_te, y_te):.2f} OOB={forest.oob_score_:.2f}")
imp = permutation_importance(forest, X_te, y_te, n_repeats=10, random_state=0)
top = np.argsort(imp.importances_mean)[::-1][:4]
print("Most important features (permutation):", [f"x{i}" for i in top])
print("Soft vote of the worked example:", np.mean([0.30, 0.40, 0.45, 0.70, 0.80]))Output
Single tree train=1.00 test=0.83 Forest 10 trees train=0.99 test=0.85 OOB=0.80 Forest 100 trees train=1.00 test=0.86 OOB=0.85 Forest 300 trees train=1.00 test=0.87 OOB=0.87 Most important features (permutation): ['x1', 'x6', 'x5', 'x9'] Soft vote of the worked example: 0.53
Try it yourself
Train a model and test it
Train on the examples, then test on new ones. Notice how a single rule can be fooled, and imagine many models voting instead.
Bagging or boosting?
Which description fits each?
Usually uses shallow trees
Mainly reduces variance
Each new tree focuses on previous mistakes
Trees are trained independently, in parallel
Common mistakes
- Trusting impurity-based importance for features with many unique values.
- Expecting a forest to predict beyond the range of training targets.
- Using too few trees, so results change from run to run.
Check yourself
Summary
- Ensembles combine models; random forests average many decorrelated trees.
- Bagging and random features reduce variance; OOB gives a free accuracy estimate.
- Forests need little tuning and no scaling, but cannot extrapolate.
Finished reading?
Sources: Leo Breiman (2001), Random Forests, Machine Learning journal · James, Witten, Hastie and Tibshirani, An Introduction to Statistical Learning (free book)